Skip to main content
Skip table of contents

Upgrade the LogRhythm Data Indexer

Upgrade a Single-Node Cluster

The script used to install or upgrade a Data Indexer cluster requires the use of a hosts file to assign a hostname to each Data Indexer node. If you used a hosts file when you first installed the Data Indexer or during a previous upgrade, point to the same file during the upgrade. If you have not previously used a hosts file, you need to create the file for the upgrade.

You must run the upgrade on each cluster. Run it on the same machine where you ran the original installer. If you have more than one node in your cluster, follow the instructions in the Multi-Node section.

Before starting the Data Indexer installation or upgrade, ensure that firewalld is running on all cluster nodes. To do this, log on to each node and run: sudo systemctl start firewalld

  1. Log on to your Indexer appliance or server as logrhythm.
  2. Change to the /home/logrhythm/Soft directory where you copied the updated installation or upgrade script.

  3. If you need to create a hosts file, use vi to create a file in /home/logrhythm/Soft called hosts.

    If you are creating a new file, ensure that you specify the current Data Indexer hostname.

    The hosts file must follow a defined pattern of {IPv4 address}, {hostname}, {boxtype}(optional) on each line. You must separate the address and hostname with a space. The file might look like the following:

    10.1.23.91 LRLinux1 hot
    If you do not specify a boxtype here, it will assume it is a hot node.  This means the warm node configuration may be lost if you do not update the hosts file prior to running the upgrade.

    Do not use fully qualified domain names for Indexer hosts. For example, use only LRLinux1 instead of LRLinux1.myorg.com.

    The following command sequence illustrates how to create and modify a file with vi:

    1. To create the hosts file and open for editing, type vi hosts.
    2. To enter INSERT mode, type i.
    3. Enter the IPv4 address, hostname to use for the Indexer, and box type, separated by a space.
    4. Press Esc.
    5. To exit and save your hosts file type: :wq
  4. To install DX and make the machine accessible without a password, download the DataIndexerLinux.zip file from the Documentation & Downloads section of the LogRhythm Community, extract the PreInstall.sh file to /home/logrhythm and execute the script.

    This cannot be run as sudo or the DX Installer will fail.
    CODE
     sh ./PreInstall.sh
  5. Run the installer with the hosts file argument:

    CODE
    sudo sh LRDataIndexer-<version>.x86_64.run --hosts <absolute path to .hosts file> --plan /home/logrhythm/Soft/plan.yml

    Press Tab after starting to type out the installer name, and the filename autocompletes for you.

  6. If prompted for the SSH password, enter the password for the logrhythm user.
    The script installs or upgrades the Data Indexer.

    This process may take up to 10 minutes.

    When the installation or upgrade is complete, a confirmation message appears.

  7. Check the status of services by typing sudo systemctl at the prompt, and then look for failed services.

    If the installation or upgrade fails with the error — failed to connect to the firewalld daemon — ensure that firewalld is running on all cluster nodes and start this procedure again. To do this, log in to each node and run the following command: sudo systemctl start firewalld

    Once the cluster restarts, there will be a short period of downtime as the DX update finalizes.

Upgrade a Multi-Node Cluster

The script used to install or upgrade a Data Indexer cluster requires the use of a hosts file to assign a hostname to each Data Indexer node. If you used a hosts file when you first installed the Data Indexer or during a previous upgrade, point to the same file during the upgrade. If you have not previously used a hosts file, you need to create the file for the upgrade.

You only need to run the upgrade on one node of each cluster, the package installer installs a Data Indexer on each node. Run it on the same machine where you ran the original installer. 

Before starting the Data Indexer installation or upgrade, ensure that firewalld is running on all cluster nodes. To do this, log in to each node and run the following command: sudo systemctl start firewalld

  1. Log on to your Indexer appliance or server as logrhythm.
  2. Change to the /home/logrhythm/Soft directory where you copied the script.

    You should have a file named hosts in the /home/logrhythm/Soft directory that was used during the original installation. The hosts file must follow a defined pattern of {IPv4 address}, {hostname}, {boxtype}(optional) on each line. You must separate the address and hostname with a space.
    The contents of the file might look like the following:

    10.1.23.65 LRLinux1 hot
    10.1.23.67 LRLinux2 warm
    10.1.23.91 LRLinux3 warm
    The box type parameter is optional in the hosts file, if you do not specify a boxtype here, it will assume it is a hot node. This means the warm node configuration may be lost if you do not update the hosts file prior to running the upgrade.
  3. If you need to create a hosts file, use vi to create a file in /home/logrhythm/Soft called hosts.

    Do not use fully qualified domain names for Indexer hosts. For example, use only LRLinux1 instead of LRLinux1.myorg.com.

    The following command sequence illustrates how to create and modify a file with vi:

    1. To create the hosts file and open for editing, type vi hosts.
    2. To enter INSERT mode, type i.
    3. Enter the IPv4 address, the hostname to use for the Indexer, and the box type, separated by spaces.
    4. Press Esc.
    5. To exit and save your hosts file type :wq.
  4. To install DX and make the machine accessible without a password, download the DataIndexerLinux.zip file from the Documentation & Downloads section of the LogRhythm Community, extract the the PreInstall.sh file to /home/logrhythm and execute the script.

    This cannot be run as sudo or the DX Installer will fail.
    CODE
    sh ./PreInstall.sh
    If there are any changes in the plan file (for example, altering hosts or version upgrades), you must copy the new plan.yml file (and remove the old plan.yml file) at /home/logrhythm/Soft.
  5. Run the installer using the original or updated hosts file:

    CODE
    sudo sh LRDataIndexer-<version>.x86_64.run --hosts <absolute path to .hosts file> --plan /home/logrhythm/plan.yml

    Press Tab after starting to type out the installer name, and the filename autocompletes for you.

  6. If prompted for the SSH password, enter the password for the logrhythm user.
    The script installs or upgrades the Data Indexer on each of the DX machines.

    This process may take up to 30 minutes.

    When the installation or upgrade is complete, a confirmation message appears.

  7. Check the status of services by typing sudo systemctl at the prompt, looking for “failed” services.

    If the installation or upgrade fails with the error — failed to connect to the firewalld daemon — ensure that firewalld is running on all cluster nodes and start the installation again. To do this, log in to each node and run the following command: sudo systemctl start firewalld

    Once the cluster restarts, there will be a short period of downtime as the DX update finalizes.

(Optional) Upgrade Using the Data Indexer Node Installer

The LogRhythm Data Indexer (LRDX) Node Installer is available to users that have purchased a DX7500 or DX7600. The installer leverages the resources on the large boxes to improve the indexing and TTL performance by adding a second Elasticsearch instance to each machine. 

The LRDX Node Installer installs and adds the second instance of Elasticsearch to the DX cluster on each DX host.

The LRDX Node Installer is needed to hit the specified performance numbers for the DX7500/DX7600.

Prerequisites

A CPU core of at least 50 and 124 GB of RAM are required for the LRDX Node Installer to run.

Upgrade an Existing DX 7500/7600

To upgrade a system running multiple nodes:

  1. Upgrade the LogRhythm deployment using the standard upgrade documentation for the version of software you are deploying. For more information, see Upgrade a LogRhythm Deployment.

  2. Connect to the Data Indexer system as a LogRhythm user.
  3. If you have not already, download the LRDataIndexer-.<version>.x86_64.run package installer to the Logrhythm user’s home directory on one of your Data Indexer appliances (for example, /home/logrhythm/Soft).

  4. Run the LRDataIndexer with the host file created in the initial install:

    CODE
    sudo sh LRDataIndexer-<version>.x86_64.run --hosts /home/logrhythm/Soft/hosts --plan /home/logrhythm/plan.yml

    The hosts file must follow a defined pattern of {IPv4 address}, {hostname}, {boxtype}(mandatory) on each line. The file might look like the following:

    10.1.23.91 LRLinux1 hot
    The box type parameter is mandatory in the hosts file, if not designated the installer will fail with a missing parameter error.
  5. Run the LRDX Node Installer with the host file:

    CODE
    sudo sh <installer> --hosts /home/logrhythm/Soft/hosts --add
    Until LR 7.5.0, it was not required to run node installer to upgrade, running the latest DX was sufficient to upgrade.
  6. When prompted for the SSH password, enter the password for the LogRhythm user.

Validate the Linux Indexer Upgrade

Validate the Data Indexer

To validate a successful upgrade of the Linux Indexer, check the following logs in /var/log/persistent:

  • ansible.log echoes console output from the upgrade, and should end with details about the number of components that upgraded successfully, as well as any issues (unreachable or failed)
  • logrhythm-node-install.sh.log lists all components that were installed or updated, along with current versions
  • logrhythm-cluster-install.sh.log should end with a message stating that the Indexer was successfully installed

Additionally, you can issue the following command and verify the installed version of various LogRhythm services, tools, and libraries, as well as third party tools:

CODE
sudo yum list installed | grep -i logrhythm
  1. Verify that the following LogRhythm services are at the same version as the main installer version:
    • Bulldozer
    • Carpenter
    • Columbo
    • GoMaintain
    • Transporter
    • Watchtower
  2. Verify that the following tools/libraries have been updated to the version matching the installer name:
    • Cluster Health
    • Conductor
    • Persistent
    • Silence
    • Unique ID
    • Upgrade Checker
  3. Verify the following versions of these services and third party tools:
    • elasticsearch 6.8.3

Verify a Warm Node

Verify a Warm Node

To identify whether a warm node is working correctly after installation, perform the following:

  1. Verify Warm Node configuration:

    CODE
    curl localhost:9200/_cat/nodeattrs?v
  2. Verify Node Settings in /usr/local/logrhythm/env/es_datapath:

    CODE
    [root@DX01 env]# cat /usr/local/logrhythm/env/es_datapath
    DX_ES_PATH_DATA=/usr/local/logrhythm/db/elasticsearch/data
    DX_ES_CLUSTER_NAME=<cluster name>
    DX_ES_DISCOVERY_ZEN_PING_UNICAST_HOSTS=<IPs of eligible master nodes>
    DX_ES_DISCOVERY_ZEN_MINIMUM_MASTER_NODES=<# of eligible master nodes/2 (rounded down) +1>
    DX_ES_BOX_TYPE=warm
    DX_ES_IS_MASTER=false
  3. On each node in /usr/local/logrhythm/transporter/logs.json, verify the number of shards and replicas based on number of hot nodes:

    CODE
    "number_of_shards": "<physical hot nodes * 2>"
    "number_of_replicas": (this will be "0" for single hot node or "1" for a multi hot node cluster)
    For 2XDX, physical nodes are only used for the shard calculation. A three-node 2XDX will have six shards.
  4. Verify warm node functionality:
    1. Wait until Elasticsearch's heap moves an open index to the warm node as a closed index.
    2. Verify that GoMaintain does not throw errors when moving the index to the warm node as Closed.
    3. (Optional) Perform an investigation against a closed index on the warm node (though this step alone will not confirm that the warm node is working).

Convert a Hot Node to a Warm Node

When making the decision to convert existing hot nodes to warm nodes, keep in mind the following rules that govern cluster topology.

  • You must have at least 1 hot node on Linux (You can't add warm node to a Windows DX on an XM)
  • You cannot have 2 hot nodes (You can have 1 or 3-10 hot nodes)

Make note of which machines (IPs) you would like to convert to warm nodes. 

For each of the machines that you would like to change to warm, perform the following:

  1. In order to access the environmental variables you will need to be running the session as root:

    CODE
    sudo su
  2. For Hot Nodes:

    CODE
    [root@DX01 env]# sudo vi /usr/local/logrhythm/env/es_datapath
    DX_ES_PATH_DATA=/usr/local/logrhythm/db/elasticsearch/data
    DX_ES_CLUSTER_NAME=<cluster name>
    DX_ES_DISCOVERY_ZEN_PING_UNICAST_HOSTS=<IPs of eligible master nodes>
    DX_ES_DISCOVERY_ZEN_MINIMUM_MASTER_NODES=<# of eligible master nodes/2 (rounded down) +1>
    DX_ES_BOX_TYPE=hot
    DX_ES_IS_MASTER=true
  3. For Warm Nodes

    CODE
    [root@DX01 env]# sudo vi /usr/local/logrhythm/env/es_datapath
    DX_ES_PATH_DATA=/usr/local/logrhythm/db/elasticsearch/data
    DX_ES_CLUSTER_NAME=<cluster name>
    DX_ES_DISCOVERY_ZEN_PING_UNICAST_HOSTS=<IPs of eligible master nodes>
    DX_ES_DISCOVERY_ZEN_MINIMUM_MASTER_NODES=<# of eligible master nodes/2 (rounded down) +1>
    DX_ES_BOX_TYPE=warm
    DX_ES_IS_MASTER=false 
  4. Run:

    CODE
    sudo systemctl restart elasticsearch
  5. Verify your cluster topology was applied correctly: 

    CODE
    curl localhost:9200/_cat/nodeattrs?v
  6. On each node edit /usr/local/logrhythm/transporter/logs.json and change the number of shards and replicas based on the number of hot nodes:

    CODE
    "number_of_shards": "<physical hot nodes * 2>"
    "number_of_replicas": (this will be "0" for single hot node or "1" for a multi hot node cluster)
    For 2XDX, you only use the physical nodes for the shard calculation.  A 3 node 2XDX will have 6 shards.
  7. To make things faster you can use sed to update the /usr/local/logrhythm/transporter/logs.json file and grep to check that change has been made:

    CODE
    grep -i shards /usr/local/logrhythm/transporter/logs.json
    
    sudo sed -i 's/"<current shard value>"/"<new shard value>"/g' /usr/local/logrhythm/transporter/logs.json
    
    grep -i shards /usr/local/logrhythm/transporter/logs.json
  8. After updating that file for each node, restart the transporter on all nodes:

    CODE
    sudo systemctl restart transporter
  9. If the new topology has a single hot node:
    1. Remove the replicas on current indices:

      CODE
      for index in $(curl -XGET http://localhost:9200/_cat/indices | grep open | sort -rk 3 | awk '{print $3}'); do 
      if [ ! -z "$index" ]; then 
      echo Changing index ${index} 
      curl -XPUT "http://localhost:9200/${index}/_settings" -H 'Content-Type: application/json' -d' { "index" : {"number_of_replicas" : 0} }' 
      fi 
      done


    2.  (Optional) Close indices which are warm and open:

      CODE
      for index in $(curl -XGET http://localhost:9200/_cat/indices | grep open | sort -rk 3| awk '{print $3}'); do 
      if [ ! -z "$index" ]; then 
      type=$(curl -XGET "localhost:9200/${index}/_settings?pretty " | grep box_type | awk '{print $3}'); 
      if [ $type = "\"warm\"" ]; then 
      echo Closing open index on warm node $index 
      curl -XPOST "http://localhost:9200/${index}/_close" 
      fi 
      fi 
      done


    3. (Optional) Move manually closed indices to the Warm Node:

      CODE
      curl -X POST "http://localhost:9200/logs-yyyy-mm-dd/_open"
      
      curl -X POST "http://localhost:9200/logs-yyyy-mm-dd/_open" curl -XPUT "http://localhost:9200/logs-yyyy-mm-dd/_settings" -H 'Content-Type: application/json' -d' { "index" : {"number_of_replicas" : 0} }'
      
      curl -XPUT "http://localhost:9200/logs-yyyy-mm-dd/_settings" -H 'Content-Type: application/json' -d' { "index.routing.allocation.require.box_type": "warm"}'
      
JavaScript errors detected

Please note, these errors can depend on your browser setup.

If this problem persists, please contact our support.