Upgrade CentOS 7 to Rocky Linux 9 in IAAS/Public Cloud

This guide outlines the high-level steps that are required in the event your Data Indexers are hosted in an Infrastructure-as-a-service or Public Cloud environment such as AWS, GCP, or Azure. The specific details or commands will vary depending on how your specific environment was deployed. Due to the very large number of variations in which how IAAS or Public Cloud deployments can be built, Exabeam does not have the ability to provide a precise step-by-step guide. This guide can be used as a roadmap or a reference point for your upgrade procedure.

If you are not confident in the areas of Linux OS, Storage Configuration, and DevOps deployment methods with your IAAS/Public Cloud, it is recommended to reach out to our Exabeam Professional Services team for assistance.

Before You Begin

Record the existing details of your CentOS 7 instance:

fstab - to identify disk mount points.
lsblk - to identify the block disks and how they are configured. Due to limits with some public cloud providers, you may have LVMs configured.
vgdisplay - to list any LVMs that are configured.
ipaddr - to record the IP address of the host. This same IP should be used on the new host.
hostname - to record the hostname of the host. This same hostname should be used on the new host.
Copy the hosts and plan files from the /home/logrhythm/Soft directory.

Upgrade Instructions

To upgrade CentOS7/RHEL7 to Rocky Linux 9, do the following:

Shutdown the existing Data Indexer instance using command line.
Create a new Data Indexer instance using the latest Rocky Linux 9 (official) instance type matching the specs of your existing server.
Remap the storage volumes used on your old instance to the new instance for the /usr/local/logrhythm mount point.
Start the new instance.
Reconfigure the storage for the /usr/local/logrhythm mount point as it was on the old instance.

This process will vary from environment to environment. The recommended design is using an LVM stripe across all data disks of equal size/type, and mounting that virtual disk group to /usr/local/logrhythm.

Install the following pre-requisite packages:

CODE

# sudo yum install firewalld
# sudo yum install sshpass
# sudo yum install chrony
# sudo yum install tar

(If using multi-node clusters) Remove the existing known host keys from the CentOS 7 installation.
The IP entered in the command should be the IP address of the machine that was just upgraded.

This step must be performed on ALL nodes prior to running PreInstall.

CODE

[logrhythm@localhost Soft]$ ssh-keygen -R 10.7.10.139
# Host 10.7.10.139 found: line 1
# Host 10.7.10.139 found: line 2
# Host 10.7.10.139 found: line 3
/home/logrhythm/.ssh/known_hosts updated.
Original contents retained as /home/logrhythm/.ssh/known_hosts.old

Copy back the hosts and plan files to the /home/logrhythm/Soft directory and run the LogRhythm DX PreInstall script.
Run the Data Indexer Installer with the --force and --es-cluster-name options to reinstall the services and restore OS specific configurations.

If you have a multi-node cluster, it is recommended you re-run the Data Indexer Installer from the same node you originally installed it.

If you are stepping through OS upgrades, you will have to re-run the DX installer after each node has been upgraded (i.e., a five-node DX cluster requires the DX installer to be run five times, once on each node, as you step through upgrading the OS one-by-one).

Ensure the DX Cluster Name matches your cluster name from prior to the OS upgrade.

Changing the DX Cluster Name could result in data loss.

Run the following command to start the remaining DX services when the install is finished.

CODE

sudo sh ./LRDataIndexer-13.59.0.0.x86_64.run --hosts /home/logrhythm/Soft/hosts --plan /home/logrhythm/Soft/plan.yml --es-cluster-name LinuxDX --force

Run the following commands to update service ownership to the following directories in order for the LogRhythm Common services to function:

CODE

sudo chown -R LogRhythmAPIGateway.LogRhythmAPIGateway /usr/local/logrhythm/LogRhythmAPIGateway
sudo chown -R LogRhythmMetricsCollection.LogRhythmMetricsCollection /usr/local/logrhythm/LogRhythmMetricsCollection
sudo chown -R LogRhythmServiceRegistry.LogRhythmServiceRegistry /usr/local/logrhythm/LogRhythmServiceRegistry
sudo systemctl restart LogRhythmAPIGateway LogRhythmMetricsCollection LogRhythmServiceRegistry

For DX7x00 models that have two Elasticsearch processes installed, you need to restore permissions to the secondary data directory after the DX installer is run, and also re-run the Nodes Installer to re-install the second Elasticsearch instance using the following commands:

CODE

[logrhythm@localhost elasticsearch]$ sudo chown -R elasticsearch:elasticsearch /usr/local/logrhythm/elasticsearch-data
[logrhythm@localhost elasticsearch]$ sudo sh ./LRDXNodeInstaller-13.64.0.1.x86_64.run --hosts /home/logrhythm/Soft/hosts --add

Check the Cluster Health status using the following commands, ensuring to also monitor the “active_shards_percent_as_number” line.
The Health status should be “Green.”

Multi-node clusters or clusters with many indexes may take some time to transition from Red to Yellow to Green.

CODE

[logrhythm@localhost elasticsearch]$ curl localhost:9200/_cluster/health?pretty
{
  "cluster_name" : "LinuxDX",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 1,
  "number_of_data_nodes" : 1,
  "active_primary_shards" : 18,
  "active_shards" : 18,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0

Check your indexes to validate that historical indexes from prior to the upgrade are present.
In the example below, the output has only one day of indexes; however, typically many days should be present.

CODE

[logrhythm@localhost elasticsearch]$ curl localhost:9200/_cat/indices/logs-*
green open logs-2023-10-23 nRhq6zUXStCK1RUSrmU8kg 2 0 6384661 2563 2gb 2gb

Validate all spooling has cleared from the "DXReliablePersist" folder in the Data Processor State directory before proceeding to upgrade any further nodes.
Reinstall the LogRhythm Agent if it was previously installed and configured using the steps outlined at Install a System Monitor on UNIX/Linux.