Upgrade the LogRhythm Data Indexer

Configure a Proxy Connection for Indexer Upgrades

If your Linux Data Indexer sits behind a proxy server, you need to add the proxy address and optional username and password to the yum configuration file on the Indexer from which you are running the upgrade.

To configure proxy options in yum.conf:

1. Log on to your Indexer appliance or server as logrhythm.

2. To open the file for editing, type: 

   CODE

   sudo vi /etc/yum.conf 

3. To enter INSERT mode, type i.

4. Add the following lines to the file:

   proxy=<proxyURL:port>

   proxy_username=<username>

   proxy_password=<password>

   EXAMPLE

   proxy=http://my.proxyaddress.com:9999/

   proxy_username=myloginID

   proxy_password=mypassword

5. Press Esc.
6. To exit and save yum.conf type :wq

Configure Upgrades Without Internet Access (Dark Sites)

If your Linux Data Indexer does not have access to the Internet (for example, in a restricted environment or at a dark site), you may need to modify CentOS-Base.repo so that repositories are skipped if they are unavailable.

CentOS-Base.repo contains the base, updates, extras, and centosplus repositories. By default, updates to centosplus are disabled (i.e., enabled is set to 0). For base, updates, and extras, you will need to add a line that will skip updates if the repo is unavailable.

If you are upgrading a multi-node cluster, you only need to modify CentOS-Base.repo on the node from which you will be running the upgrade.

To configure repository options in CentOS-Base.repo:

1. Log in to your Indexer appliance or server as logrhythm.

2. To open the file for editing, type:

   CODE

   sudo vi /etc/yum.repos.d/CentOS-Base.repo

3. To enter INSERT mode, type i.

4. Within each of the three repository sections — base, updates, and extras — add the following line:

   CODE

   skip_if_unavailable=true

5. Press Esc.
6. To exit and save CentOS-Base.repo type :wq

Upgrade a Single-Node Cluster

The script used to install or upgrade a Data Indexer cluster requires the use of a hosts file to assign a hostname to each Data Indexer node. If you used a hosts file when you first installed the Data Indexer or during a previous upgrade, point to the same file during the upgrade. If you have not previously used a hosts file, you need to create the file for the upgrade.

You must run the upgrade on each cluster. Run it on the same machine where you ran the original installer. If you have more than one node in your cluster, follow the instructions in the Multi-Node section.

Before starting the Data Indexer installation or upgrade, ensure that firewalld is running on all cluster nodes. To do this, log on to each node and run: sudo systemctl start firewalld

1. Log on to your Indexer appliance or server as logrhythm.

2. Change to the /home/logrhythm/Soft directory where you copied the updated installation or upgrade script.

3. If you need to create a hosts file, use vi to create a file in /home/logrhythm/Soft called hosts.

   

   If you are creating a new file, ensure that you specify the current Data Indexer hostname.

   The hosts file must follow a defined pattern of {IPv4 address}, {hostname}, {boxtype}(optional) on each line. You must separate the address and hostname with a space. The file might look like the following:

   10.1.23.91 LRLinux1 hot

   

   If you do not specify a boxtype here, it will assume it is a hot node.  This means the warm node configuration may be lost if you do not update the hosts file prior to running the upgrade.
   

   Do not use fully qualified domain names for Indexer hosts. For example, use only LRLinux1 instead of LRLinux1.myorg.com.

   The following command sequence illustrates how to create and modify a file with vi:

   1. To create the hosts file and open for editing, type vi hosts.
   2. To enter INSERT mode, type i.
   3. Enter the IPv4 address, hostname to use for the Indexer, and box type, separated by a space.
   4. Press Esc.
   5. To exit and save your hosts file type: :wq

4. To install DX and make the machine accessible without a password, download the DataIndexerLinux.zip file from the Documentation & Downloads section of the LogRhythm Community, extract the PreInstall.sh file to /home/logrhythm and execute the script.

   

   This cannot be run as sudo or the DX Installer will fail.
   CODE

    sh ./PreInstall.sh

5. Run the installer with the hosts file argument:

   CODE

   sudo sh LRDataIndexer-<version>.centos.x86_64.run --hosts <absolute path to .hosts file> --plan /home/logrhythm/Soft/plan.yml

   Press Tab after starting to type out the installer name, and the filename autocompletes for you.

6. If prompted for the SSH password, enter the password for the logrhythm user.
   The script installs or upgrades the Data Indexer.

   

   This process may take up to 10 minutes.

   When the installation or upgrade is complete, a confirmation message appears.

7. Check the status of services by typing sudo systemctl at the prompt, and then look for failed services.

   

   If the installation or upgrade fails with the error — failed to connect to the firewalld daemon — ensure that firewalld is running on all cluster nodes and start this procedure again. To do this, log in to each node and run the following command: sudo systemctl start firewalld
   

   Once the cluster restarts, there will be a short period of downtime as the DX update finalizes.

Upgrade a Multi-Node Cluster

The script used to install or upgrade a Data Indexer cluster requires the use of a hosts file to assign a hostname to each Data Indexer node. If you used a hosts file when you first installed the Data Indexer or during a previous upgrade, point to the same file during the upgrade. If you have not previously used a hosts file, you need to create the file for the upgrade.

You only need to run the upgrade on one node of each cluster, the package installer installs a Data Indexer on each node. Run it on the same machine where you ran the original installer. 

Before starting the Data Indexer installation or upgrade, ensure that firewalld is running on all cluster nodes. To do this, log in to each node and run the following command: sudo systemctl start firewalld

1. Log on to your Indexer appliance or server as logrhythm.

2. Change to the /home/logrhythm/Soft directory where you copied the script.

   You should have a file named hosts in the /home/logrhythm/Soft directory that was used during the original installation. The hosts file must follow a defined pattern of {IPv4 address}, {hostname}, {boxtype}(optional) on each line. You must separate the address and hostname with a space.
   The contents of the file might look like the following:

   10.1.23.65 LRLinux1 hot
   10.1.23.67 LRLinux2 warm
   10.1.23.91 LRLinux3 warm

   

   The box type parameter is optional in the hosts file, if you do not specify a boxtype here, it will assume it is a hot node. This means the warm node configuration may be lost if you do not update the hosts file prior to running the upgrade.

3. If you need to create a hosts file, use vi to create a file in /home/logrhythm/Soft called hosts.

   

   Do not use fully qualified domain names for Indexer hosts. For example, use only LRLinux1 instead of LRLinux1.myorg.com.

   The following command sequence illustrates how to create and modify a file with vi:

   1. To create the hosts file and open for editing, type vi hosts.
   2. To enter INSERT mode, type i.
   3. Enter the IPv4 address, the hostname to use for the Indexer, and the box type, separated by spaces.
   4. Press Esc.
   5. To exit and save your hosts file type :wq.

4. To install DX and make the machine accessible without a password, download the DataIndexerLinux.zip file from the Documentation & Downloads section of the LogRhythm Community, extract the the PreInstall.sh file to /home/logrhythm and execute the script.

   

   This cannot be run as sudo or the DX Installer will fail.
   CODE

   sh ./PreInstall.sh

   

   If there are any changes in the plan file, you must copy the new plan file at /home/logrhythm/Soft.

5. Run the installer using the original or updated hosts file:

   CODE

   sudo sh LRDataIndexer-<version>.centos.x86_64.run --hosts <absolute path to .hosts file> --plan /home/logrhythm/plan.yml

   Press Tab after starting to type out the installer name, and the filename autocompletes for you.

6. If prompted for the SSH password, enter the password for the logrhythm user.
   The script installs or upgrades the Data Indexer on each of the DX machines.

   

   This process may take up to 30 minutes.

   When the installation or upgrade is complete, a confirmation message appears.

7. Check the status of services by typing sudo systemctl at the prompt, looking for “failed” services.

   

   If the installation or upgrade fails with the error — failed to connect to the firewalld daemon — ensure that firewalld is running on all cluster nodes and start the installation again. To do this, log in to each node and run the following command: sudo systemctl start firewalld
   

   Once the cluster restarts, there will be a short period of downtime as the DX update finalizes.

(Optional) Upgrade Using the Data Indexer Node Installer

The LogRhythm Data Indexer (LRDX) Node Installer is available to users that have purchased a DX 7500. The installer leverages the resources on the DX 7500 to improve the indexing and TTL performance by adding a second Elasticsearch instance to each DX 7500. 

The LRDX Node Installer installs and adds the second instance of Elasticsearch to the DX cluster on each DX host.

The LRDX Node Installer is needed to hit the specified performance numbers for the DX 7500.

Prerequisites

A CPU core of at least 50 and 124 GB of RAM are required for the LRDX Node Installer to run.

Upgrade an Existing DX 7500

To upgrade a system running multiple nodes:

1. Upgrade the LogRhythm deployment using the standard upgrade documentation for the version of software you are deploying. For more information, see Upgrade a LogRhythm Deployment.

2. Connect to the Data Indexer system as a LogRhythm user.

3. If you have not already, download the LRDataIndexer-.<version>.centos.x86_64.run package installer to the Logrhythm user’s home directory on one of your Data Indexer appliances (for example, /home/logrhythm/Soft).

4. Run the LRDataIndexer with the host file created in the initial install:

   CODE

   sudo sh LRDataIndexer-<version>.centos.x86_64.run --hosts /home/logrhythm/Soft/hosts --plan /home/logrhythm/plan.yml

   The hosts file must follow a defined pattern of {IPv4 address}, {hostname}, {boxtype}(mandatory) on each line. The file might look like the following:

   10.1.23.91 LRLinux1 hot

   

   The box type parameter is mandatory in the hosts file, if not designated the installer will fail with a —missing parameter— error.

5. Run the LRDX Node Installer with the host file:

   CODE

   sudo sh <installer> --hosts /home/logrhythm/Soft/hosts --add

   

   Until LR 7.5.0, it was not required to run node installer to upgrade, running the latest DX was sufficient to upgrade.

6. When prompted for the SSH password, enter the password for the LogRhythm user.

Validate the Linux Indexer Upgrade

sudo yum list installed | grep -i logrhythm

Verify a Warm Node

Convert a Hot Node to a Warm Node

When making the decision to convert existing hot nodes to warm nodes, keep in mind the following rules that govern cluster topology.

• You must have at least 1 hot node on Linux (You can't add warm node to a Windows DX on an XM)
• You cannot have 2 hot nodes (You can have 1 or 3-10 hot nodes)

Make note of which machines (IPs) you would like to convert to warm nodes. 

For each of the machines that you would like to change to warm, perform the following:

1. In order to access the environmental variables you will need to be running the session as root:

   CODE

   sudo su

2. For Hot Nodes:

   CODE

   [root@DX01 env]# sudo vi /usr/local/logrhythm/env/es_datapath
   DX_ES_PATH_DATA=/usr/local/logrhythm/db/elasticsearch/data
   DX_ES_CLUSTER_NAME=<cluster name>
   DX_ES_DISCOVERY_ZEN_PING_UNICAST_HOSTS=<IPs of eligible master nodes>
   DX_ES_DISCOVERY_ZEN_MINIMUM_MASTER_NODES=<# of eligible master nodes/2 (rounded down) +1>
   DX_ES_BOX_TYPE=hot
   DX_ES_IS_MASTER=true

3. For Warm Nodes

   CODE

   [root@DX01 env]# sudo vi /usr/local/logrhythm/env/es_datapath
   DX_ES_PATH_DATA=/usr/local/logrhythm/db/elasticsearch/data
   DX_ES_CLUSTER_NAME=<cluster name>
   DX_ES_DISCOVERY_ZEN_PING_UNICAST_HOSTS=<IPs of eligible master nodes>
   DX_ES_DISCOVERY_ZEN_MINIMUM_MASTER_NODES=<# of eligible master nodes/2 (rounded down) +1>
   DX_ES_BOX_TYPE=warm
   DX_ES_IS_MASTER=false 

4. Run:

   CODE

   sudo systemctl restart elasticsearch

5. Verify your cluster topology was applied correctly: 
   

   CODE

   curl localhost:9200/_cat/nodeattrs?v

6. On each node edit /usr/local/logrhythm/transporter/logs.json and change the number of shards and replicas based on the number of hot nodes:

   CODE

   "number_of_shards": "<physical hot nodes * 2>"
   "number_of_replicas": (this will be "0" for single hot node or "1" for a multi hot node cluster)

   

   For 2XDX, you only use the physical nodes for the shard calculation.  A 3 node 2XDX will have 6 shards.

7. To make things faster you can use sed to update the /usr/local/logrhythm/transporter/logs.json file and grep to check that change has been made:

   CODE

   grep -i shards /usr/local/logrhythm/transporter/logs.json
   
   sudo sed -i 's/"<current shard value>"/"<new shard value>"/g' /usr/local/logrhythm/transporter/logs.json
   
   grep -i shards /usr/local/logrhythm/transporter/logs.json

8. After updating that file for each node, restart the transporter on all nodes:

   CODE

   sudo systemctl restart transporter

9. If the new topology has a single hot node:
   

   1. Remove the replicas on current indices:
      

      CODE

      for index in $(curl -XGET http://localhost:9200/_cat/indices | grep open | sort -rk 3 | awk '{print $3}'); do 
      if [ ! -z "$index" ]; then 
      echo Changing index ${index} 
      curl -XPUT "http://localhost:9200/${index}/_settings" -H 'Content-Type: application/json' -d' { "index" : {"number_of_replicas" : 0} }' 
      fi 
      done

      
      

   2.  (Optional) Close indices which are warm and open:
      

      CODE

      for index in $(curl -XGET http://localhost:9200/_cat/indices | grep open | sort -rk 3| awk '{print $3}'); do 
      if [ ! -z "$index" ]; then 
      type=$(curl -XGET "localhost:9200/${index}/_settings?pretty " | grep box_type | awk '{print $3}'); 
      if [ $type = "\"warm\"" ]; then 
      echo Closing open index on warm node $index 
      curl -XPOST "http://localhost:9200/${index}/_close" 
      fi 
      fi 
      done

      
      

   3. (Optional) Move manually closed indices to the Warm Node:
      

      CODE

      curl -X POST "http://localhost:9200/logs-yyyy-mm-dd/_open"
      
      curl -X POST "http://localhost:9200/logs-yyyy-mm-dd/_open" curl -XPUT "http://localhost:9200/logs-yyyy-mm-dd/_settings" -H 'Content-Type: application/json' -d' { "index" : {"number_of_replicas" : 0} }'
      
      curl -XPUT "http://localhost:9200/logs-yyyy-mm-dd/_settings" -H 'Content-Type: application/json' -d' { "index.routing.allocation.require.box_type": "warm"}'