Upgrade the LogRhythm Data Indexer
Configure a Proxy Connection for Indexer Upgrades
If your Linux Data Indexer sits behind a proxy server, you need to add the proxy address and optional username and password to the yum configuration file on the Indexer from which you are running the upgrade.
To configure proxy options in yum.conf:
- Log on to your Indexer appliance or server as logrhythm.
To open the file for editing, type:
CODEsudo vi /etc/yum.conf
- To enter INSERT mode, type i.
Add the following lines to the file:
proxy=<proxyURL:port>
proxy_username=<username>
proxy_password=<password>
EXAMPLE
proxy=http://my.proxyaddress.com:9999/
proxy_username=myloginID
proxy_password=mypassword
- Press Esc.
- To exit and save yum.conf type
:wq
Configure Upgrades Without Internet Access (Dark Sites)
If your Linux Data Indexer does not have access to the Internet (for example, in a restricted environment or at a dark site), you may need to modify CentOS-Base.repo so that repositories are skipped if they are unavailable.
CentOS-Base.repo contains the base, updates, extras, and centosplus repositories. By default, updates to centosplus are disabled (i.e., enabled is set to 0). For base, updates, and extras, you will need to add a line that will skip updates if the repo is unavailable.
To configure repository options in CentOS-Base.repo:
- Log in to your Indexer appliance or server as logrhythm.
To open the file for editing, type:
CODEsudo vi /etc/yum.repos.d/CentOS-Base.repo
- To enter INSERT mode, type i.
Within each of the three repository sections — base, updates, and extras — add the following line:
CODEskip_if_unavailable=true
- Press Esc.
- To exit and save CentOS-Base.repo type
:wq
Upgrade a Single-Node Cluster
The script used to install or upgrade a Data Indexer cluster requires the use of a hosts file to assign a hostname to each Data Indexer node. If you used a hosts file when you first installed the Data Indexer or during a previous upgrade, point to the same file during the upgrade. If you have not previously used a hosts file, you need to create the file for the upgrade.
Before starting the Data Indexer installation or upgrade, ensure that firewalld is running on all cluster nodes. To do this, log on to each node and run: sudo systemctl start firewalld
- Log on to your Indexer appliance or server as logrhythm.
Change to the /home/logrhythm/Soft directory where you copied the updated installation or upgrade script.
If you need to create a hosts file, use
vi
to create a file in /home/logrhythm/Soft called hosts.If you are creating a new file, ensure that you specify the current Data Indexer hostname.The hosts file must follow a defined pattern of {IPv4 address}, {hostname}, {boxtype}(optional) on each line. You must separate the address and hostname with a space. The file might look like the following:
10.1.23.91 LRLinux1 hot
If you do not specify a boxtype here, it will assume it is a hot node. This means the warm node configuration may be lost if you do not update the hosts file prior to running the upgrade.Do not use fully qualified domain names for Indexer hosts. For example, use only LRLinux1 instead of LRLinux1.myorg.com.
The following command sequence illustrates how to create and modify a file with vi:
- To create the hosts file and open for editing, type vi hosts.
- To enter INSERT mode, type i.
- Enter the IPv4 address, hostname to use for the Indexer, and box type, separated by a space.
- Press Esc.
- To exit and save your hosts file type:
:wq
To install DX and make the machine accessible without a password, download the DataIndexerLinux.zip file from the Documentation & Downloads section of the LogRhythm Community, extract the PreInstall.sh file to /home/logrhythm and execute the script.
This cannot be run assudo
or the DX Installer will fail.CODEsh ./PreInstall.sh
Run the installer with the hosts file argument:
CODEsudo sh LRDataIndexer-<version>.centos.x86_64.run --hosts <absolute path to .hosts file> --plan /home/logrhythm/Soft/plan.yml
Press Tab after starting to type out the installer name, and the filename autocompletes for you.
If prompted for the SSH password, enter the password for the logrhythm user.
The script installs or upgrades the Data Indexer.This process may take up to 10 minutes.When the installation or upgrade is complete, a confirmation message appears.
Check the status of services by typing sudo systemctl at the prompt, and then look for failed services.
If the installation or upgrade fails with the error — failed to connect to the firewalld daemon — ensure that firewalld is running on all cluster nodes and start this procedure again. To do this, log in to each node and run the following command:sudo systemctl start firewalld
Once the cluster restarts, there will be a short period of downtime as the DX update finalizes.
Upgrade a Multi-Node Cluster
The script used to install or upgrade a Data Indexer cluster requires the use of a hosts file to assign a hostname to each Data Indexer node. If you used a hosts file when you first installed the Data Indexer or during a previous upgrade, point to the same file during the upgrade. If you have not previously used a hosts file, you need to create the file for the upgrade.
Before starting the Data Indexer installation or upgrade, ensure that firewalld is running on all cluster nodes. To do this, log in to each node and run the following command: sudo systemctl start firewalld
- Log on to your Indexer appliance or server as logrhythm.
Change to the /home/logrhythm/Soft directory where you copied the script.
You should have a file named hosts in the /home/logrhythm/Soft directory that was used during the original installation. The hosts file must follow a defined pattern of {IPv4 address}, {hostname}, {boxtype}(optional) on each line. You must separate the address and hostname with a space.
The contents of the file might look like the following:10.1.23.65 LRLinux1 hot
10.1.23.67 LRLinux2 warm
10.1.23.91 LRLinux3 warmThe box type parameter is optional in the hosts file, if you do not specify a boxtype here, it will assume it is a hot node. This means the warm node configuration may be lost if you do not update the hosts file prior to running the upgrade.If you need to create a hosts file, use
vi
to create a file in /home/logrhythm/Soft called hosts.Do not use fully qualified domain names for Indexer hosts. For example, use only LRLinux1 instead of LRLinux1.myorg.com.
The following command sequence illustrates how to create and modify a file with
vi
:- To create the hosts file and open for editing, type
vi hosts
. - To enter INSERT mode, type i.
- Enter the IPv4 address, the hostname to use for the Indexer, and the box type, separated by spaces.
- Press Esc.
- To exit and save your hosts file type
:wq
.
- To create the hosts file and open for editing, type
To install DX and make the machine accessible without a password, download the DataIndexerLinux.zip file from the Documentation & Downloads section of the LogRhythm Community, extract the the PreInstall.sh file to /home/logrhythm and execute the script.
This cannot be run assudo
or the DX Installer will fail.CODEsh ./PreInstall.sh
If there are any changes in the plan file, you must copy the new plan file at /home/logrhythm/Soft.Run the installer using the original or updated hosts file:
CODEsudo sh LRDataIndexer-<version>.centos.x86_64.run --hosts <absolute path to .hosts file> --plan /home/logrhythm/plan.yml
Press Tab after starting to type out the installer name, and the filename autocompletes for you.
If prompted for the SSH password, enter the password for the logrhythm user.
The script installs or upgrades the Data Indexer on each of the DX machines.This process may take up to 30 minutes.When the installation or upgrade is complete, a confirmation message appears.
Check the status of services by typing
sudo systemctl
at the prompt, looking for “failed” services.If the installation or upgrade fails with the error — failed to connect to the firewalld daemon — ensure that firewalld is running on all cluster nodes and start the installation again. To do this, log in to each node and run the following command:sudo systemctl start firewalld
Once the cluster restarts, there will be a short period of downtime as the DX update finalizes.
(Optional) Upgrade Using the Data Indexer Node Installer
The LogRhythm Data Indexer (LRDX) Node Installer is available to users that have purchased a DX 7500. The installer leverages the resources on the DX 7500 to improve the indexing and TTL performance by adding a second Elasticsearch instance to each DX 7500.
The LRDX Node Installer installs and adds the second instance of Elasticsearch to the DX cluster on each DX host.
The LRDX Node Installer is needed to hit the specified performance numbers for the DX 7500.
Prerequisites
A CPU core of at least 50 and 124 GB of RAM are required for the LRDX Node Installer to run.
Upgrade an Existing DX 7500
To upgrade a system running multiple nodes:
Upgrade the LogRhythm deployment using the standard upgrade documentation for the version of software you are deploying. For more information, see Upgrade a LogRhythm Deployment.
- Connect to the Data Indexer system as a LogRhythm user.
If you have not already, download the LRDataIndexer-.<version>.centos.x86_64.run package installer to the Logrhythm user’s home directory on one of your Data Indexer appliances (for example, /home/logrhythm/Soft).
Run the LRDataIndexer with the host file created in the initial install:
CODEsudo sh LRDataIndexer-<version>.centos.x86_64.run --hosts /home/logrhythm/Soft/hosts --plan /home/logrhythm/plan.yml
The hosts file must follow a defined pattern of {IPv4 address}, {hostname}, {boxtype}(mandatory) on each line. The file might look like the following:
10.1.23.91 LRLinux1 hot
The box type parameter is mandatory in the hosts file, if not designated the installer will fail with a —missing parameter— error.Run the LRDX Node Installer with the host file:
CODEsudo sh <installer> --hosts /home/logrhythm/Soft/hosts --add
Until LR 7.5.0, it was not required to run node installer to upgrade, running the latest DX was sufficient to upgrade.When prompted for the SSH password, enter the password for the LogRhythm user.
Validate the Linux Indexer Upgrade
To validate a successful upgrade of the Linux Indexer, check the following logs in /var/log/persistent:
- ansible.log echoes console output from the upgrade, and should end with details about the number of components that upgraded successfully, as well as any issues (unreachable or failed)
- logrhythm-node-install.sh.log lists all components that were installed or updated, along with current versions
- logrhythm-cluster-install.sh.log should end with a message stating that the Indexer was successfully installed
Additionally, you can issue the following command and verify the installed version of various LogRhythm services, tools, and libraries, as well as third party tools:
sudo yum list installed | grep -i logrhythm
- Verify that the following LogRhythm services are at the same version as the main installer version:
- Bulldozer
- Carpenter
- Columbo
- GoMaintain
- Transporter
- Watchtower
- Verify that the following tools/libraries have been updated to the version matching the installer name:
- Cluster Health
- Conductor
- Persistent
- Silence
- Unique ID
- Upgrade Checker
- Verify the following versions of these services and third party tools:
- elasticsearch 6.8.3
Verify a Warm Node
To identify whether a warm node is working correctly after installation, perform the following:
Verify Warm Node configuration:
CODEcurl localhost:9200/_cat/nodeattrs?v
Verify Node Settings in /usr/local/logrhythm/env/es_datapath:
CODE[root@DX01 env]# cat /usr/local/logrhythm/env/es_datapath DX_ES_PATH_DATA=/usr/local/logrhythm/db/elasticsearch/data DX_ES_CLUSTER_NAME=<cluster name> DX_ES_DISCOVERY_ZEN_PING_UNICAST_HOSTS=<IPs of eligible master nodes> DX_ES_DISCOVERY_ZEN_MINIMUM_MASTER_NODES=<# of eligible master nodes/2 (rounded down) +1> DX_ES_BOX_TYPE=warm DX_ES_IS_MASTER=false
On each node in /usr/local/logrhythm/transporter/logs.json, verify the number of shards and replicas based on number of hot nodes:
CODE"number_of_shards": "<physical hot nodes * 2>" "number_of_replicas": (this will be "0" for single hot node or "1" for a multi hot node cluster)
For 2XDX, physical nodes are only used for the shard calculation. A three-node 2XDX will have six shards.- Verify warm node functionality:
- Wait until Elasticsearch's heap moves an open index to the warm node as a closed index.
- Verify that GoMaintain does not throw errors when moving the index to the warm node as Closed.
- (Optional) Perform an investigation against a closed index on the warm node (though this step alone will not confirm that the warm node is working).
Convert a Hot Node to a Warm Node
When making the decision to convert existing hot nodes to warm nodes, keep in mind the following rules that govern cluster topology.
- You must have at least 1 hot node on Linux (You can't add warm node to a Windows DX on an XM)
- You cannot have 2 hot nodes (You can have 1 or 3-10 hot nodes)
Make note of which machines (IPs) you would like to convert to warm nodes.
For each of the machines that you would like to change to warm, perform the following:
In order to access the environmental variables you will need to be running the session as root:
CODEsudo su
For Hot Nodes:
CODE[root@DX01 env]# sudo vi /usr/local/logrhythm/env/es_datapath DX_ES_PATH_DATA=/usr/local/logrhythm/db/elasticsearch/data DX_ES_CLUSTER_NAME=<cluster name> DX_ES_DISCOVERY_ZEN_PING_UNICAST_HOSTS=<IPs of eligible master nodes> DX_ES_DISCOVERY_ZEN_MINIMUM_MASTER_NODES=<# of eligible master nodes/2 (rounded down) +1> DX_ES_BOX_TYPE=hot DX_ES_IS_MASTER=true
For Warm Nodes
CODE[root@DX01 env]# sudo vi /usr/local/logrhythm/env/es_datapath DX_ES_PATH_DATA=/usr/local/logrhythm/db/elasticsearch/data DX_ES_CLUSTER_NAME=<cluster name> DX_ES_DISCOVERY_ZEN_PING_UNICAST_HOSTS=<IPs of eligible master nodes> DX_ES_DISCOVERY_ZEN_MINIMUM_MASTER_NODES=<# of eligible master nodes/2 (rounded down) +1> DX_ES_BOX_TYPE=warm DX_ES_IS_MASTER=false
Run:
CODEsudo systemctl restart elasticsearch
Verify your cluster topology was applied correctly:
CODEcurl localhost:9200/_cat/nodeattrs?v
On each node edit /usr/local/logrhythm/transporter/logs.json and change the number of shards and replicas based on the number of hot nodes:
CODE"number_of_shards": "<physical hot nodes * 2>" "number_of_replicas": (this will be "0" for single hot node or "1" for a multi hot node cluster)
For 2XDX, you only use the physical nodes for the shard calculation. A 3 node 2XDX will have 6 shards.To make things faster you can use
sed
to update the /usr/local/logrhythm/transporter/logs.json file and grep to check that change has been made:CODEgrep -i shards /usr/local/logrhythm/transporter/logs.json sudo sed -i 's/"<current shard value>"/"<new shard value>"/g' /usr/local/logrhythm/transporter/logs.json grep -i shards /usr/local/logrhythm/transporter/logs.json
After updating that file for each node, restart the transporter on all nodes:
CODEsudo systemctl restart transporter
- If the new topology has a single hot node:
Remove the replicas on current indices:
CODEfor index in $(curl -XGET http://localhost:9200/_cat/indices | grep open | sort -rk 3 | awk '{print $3}'); do if [ ! -z "$index" ]; then echo Changing index ${index} curl -XPUT "http://localhost:9200/${index}/_settings" -H 'Content-Type: application/json' -d' { "index" : {"number_of_replicas" : 0} }' fi done
(Optional) Close indices which are warm and open:
CODEfor index in $(curl -XGET http://localhost:9200/_cat/indices | grep open | sort -rk 3| awk '{print $3}'); do if [ ! -z "$index" ]; then type=$(curl -XGET "localhost:9200/${index}/_settings?pretty " | grep box_type | awk '{print $3}'); if [ $type = "\"warm\"" ]; then echo Closing open index on warm node $index curl -XPOST "http://localhost:9200/${index}/_close" fi fi done
(Optional) Move manually closed indices to the Warm Node:
CODEcurl -X POST "http://localhost:9200/logs-yyyy-mm-dd/_open" curl -X POST "http://localhost:9200/logs-yyyy-mm-dd/_open" curl -XPUT "http://localhost:9200/logs-yyyy-mm-dd/_settings" -H 'Content-Type: application/json' -d' { "index" : {"number_of_replicas" : 0} }' curl -XPUT "http://localhost:9200/logs-yyyy-mm-dd/_settings" -H 'Content-Type: application/json' -d' { "index.routing.allocation.require.box_type": "warm"}'