High Availability Data Indexer Upgrades

Configure a Proxy Connection for Indexer Upgrades

To configure proxy options in yum.conf:

1. Log on to your Indexer appliance or server as logrhythm.

2. To open the file for editing, type: 

   sudo vi /etc/yum.conf
   

3. To enter INSERT mode, type i.

4. Add the following lines to the file:

   proxy=<proxyURL:port>
   

   proxy_username=<username>
   

   proxy_password=<password>
   

   Press Esc.

proxy=http://my.proxyaddress.com:9999/

proxy_username=myloginID

proxy_password=mypassword

5. To exit and save yum.conf type :wq

Configure Upgrades Without Internet Access (Dark Sites)

If your Linux Data Indexer does not have access to the Internet (for example, in a restricted environment or at a dark site), you may need to modify CentOS-Base.repo so that repositories are skipped if they are unavailable.

CentOS-Base.repo contains the base, updates, extras, and centosplus repositories. By default, updates to centosplus are disabled (i.e., enabled is set to 0). For base, updates, and extras, you will need to add a line that will skip updates if the repo is unavailable.

To configure repository options in CentOS-Base.repo:

1. Log in to your Indexer appliance or server as logrhythm.

2. To open the file for editing, type:

   sudo vi /etc/yum.repos.d/CentOS-Base.repo
   

3. To enter INSERT mode, type i.

4. Within each of the three repository sections — base, updates, and extras — add the following line:

   skip_if_unavailable=true
   

5. Press Esc.

6. To exit and save CentOS-Base.repo type :wq

Upgrade a Single-node Cluster

Before starting the Data Indexer upgrade, ensure that firewalld is running on all cluster nodes. To do this, log on to each node and run:

sudo systemctl start firewalld

1. Log on to your Indexer appliance or server as logrhythm.

2. Change to the /home/logrhythm/Soft directory where you copied the updated installation or upgrade script.

3. If you need to create a hosts file, use vi to create a file in /home/logrhythm/Soft called hosts.
   If you are creating a new file, ensure that you specify the current Data Indexer hostname.
   The hosts file must follow a defined pattern of {IPv4 address} {hostname} {boxtype} on each line. You must separate the address and hostname with a space. The file might look like the following for a multi-node cluster:

10.1.23.65 LRLinux1 hot
10.1.23.67 LRLinux2 warm
10.1.23.91 LRLinux3 warm

The following command sequence illustrates how to create and modify a file with vi:

1. 1. To create the hosts file and open for editing, type vi hosts.

   2. To enter INSERT mode, type i.

   3. Enter the IPv4 address, hostname to use for the Indexer, and box type, separated by a space.

   4. Press Esc.

   5. To exit and save your hosts file, type :wq

4. Download the DataIndexerLinux.zip file from the Documentation & Downloads section of the LogRhythm Community, extract the contents of the zip and place all files in /home/logrhythm/Soft.

5. Run the installer with the hosts file argument:

sudo sh LRDataIndexer-<version>.x86_64.run --hosts <absolute path to .hosts file> --plan /home/logrhythm/Soft/plan.yml

Press Tab after starting to type out the installer name, and the filename autocompletes for you.

6. The script installs or upgrades the Data Indexer.
   When the installation or upgrade is complete, a confirmation message appears.

7. Check the status of services by typing sudo systemctl at the prompt, and then look for failed services.

Validate the Linux Indexer Upgrade

To validate a successful upgrade of the Linux Indexer, check the following logs in /var/log/persistent:

• ansible.log echoes console output from the upgrade, and should end with details about the number of components that upgraded successfully, as well as any issues (unreachable or failed)

• logrhythm-node-install.sh.log lists all components that were installed or updated, along with current versions

• logrhythm-cluster-install.sh.log should end with a message stating that the Indexer was successfully installed

Additionally, you can issue the following command and verify the installed version of various LogRhythm services, tools, and libraries, as well as third party tools:

sudo yum list installed | grep -i logrhythm

1. Verify the following versions of these services and third party tools:

   • elasticsearch 7.10.2

Configure the LogRhythm Data Indexer

Configuring the Data Indexer for Windows and Linux has moved from the individual clusters, to the Configuration Manager on the Platform Manager. You can configure all data Indexers using the LogRhythm Configuration Manager installed on the Platform Manager.

• Cluster Name configuration is currently done through environment settings. Before configuring the Data Indexer on Windows, verify that the DX_OS_CLUSTER_NAME environment variable is set on both DR servers.

• LogRhythm Service Registry, LogRhythm API Gateway and LogRhythm Windows Authentication API Service must be running before opening LogRhythm Configuration Manager

• If you are configuring multiple data Indexers, all can be configured from the Primary PM as the configuration is centralized between servers.

To configure the Data Indexer:

1. Open the Configuration Manager from programs on the Platform Manager.

2. From the menu on the left, select the Data Indexers tab.

   Each installed Data Indexer has its own section that looks like this: 

   Data Indexer - Cluster Name: <ClusterName> Cluster Id: <ClusterID>

3. Verify or update the following Data Indexer settings:

Advanced View Settings:

4. Click Submit.

Automatic Maintenance

Automatic maintenance is governed by several of the above settings by the GoMaintain service. On startup, GoMaintain will continuously take samples from OpenSearch stats, including disk and heap utilization for the configured time frame. 

GoMaintain will automatically perform maintenance when High Water Mark settings are reached. Samples are taken over a period of time and analyzed before GoMaintain will take action on an index. This will depend on the Sample Interval and #Sample settings. By default, this is 60 samples, 1 every 10 seconds for a total of 10 minutes. If it is determined during that sample period that a High Water Mark setting was reached for an extended period of time, indices will be closed, deleted, or moved to warm nodes depending on the data indexer configuration. After an action is taken and completed, the sample period will begin again.

The DX monitors OpenSearch memory and DX storage capacity. GoMaintain tracks heap pressure on the nodes. If the pressure constantly crosses the threshold, GoMaintain decreases the number of days of indices by closing the index. Closing the index removes the resource needs of managing that data and relieves the heap pressure on OpenSearch. GoMaintain continues to close days until the memory is under the warning threshold, and continues to delete days based on the disk utilization setting.

Logging of configuration and results for force merge can be found in C:\Program Files\LogRhythm\DataIndexer\logs\GoMaintain.log.

GoMaintain TTL Logs (#Indices)

The default configuration is -1. This value monitors the systems resources (Heap and Disk) and automatically manages the time-to-live (TTL). You can configure a lower/static TTL by changing this number. If this number is no longer achievable due to heap consumption, the DX sends a diagnostic warning and starts closing the indices or moving them to Warm nodes if present.

Disk Utilization Limit

The disk utilization limit indicates the percentage of disk utilization that triggers maintenance. The recommended value depends on the type of disks used in your Hot DX nodes: 90% for SSD and 80% for HDD. This value triggers when maintenance starts based on disk consumption of the smallest Hot node in the cluster. Maintenance for GoMaintain will either delete the oldest index or move it from Hot to Warm tier if Warm tier is present in the cluster. The value for Disk Utilization Limit should not be set higher than 90. This value can have an impact on the ability of OpenSearch to store replica shards for the purpose of failover.

• IndexManager Disk HWM (%diskUtil) Indicates the percentage of disk utilization that triggers maintenance.

• If Warm nodes are present, the disk utilization for combined Hot and Warm nodes will be tracked separately.

If Warm nodes are present, the oldest index will be moved to the Warm node(s) if the Disk HWM is reached.

Maintenance is applied to the active repository, as well as archive repositories created by Second Look. When the Disk Usage Limit is reached, active logs are trimmed when “max indices” is reached. At this point, GoMaintain deletes completed restored repositories starting with the oldest date.

The default settings prioritize restored repositories above the active log repository. Restored archived logs are maintained while sacrificing active logs. If you want to keep your active logs and delete archives for space, set your min indices equal to your max indices. This forces the maintenance process to delete restored repositories first.

Heap Utilization Limit

• IndexManager Heap HWM (%esheap) Indicates the percentage of OpenSearch (java) heap utilization that triggers maintenance. The default is 85, which means that maintenance starts when the OpenSearch heap utilization reaches 85%.

• If the Heap HWM is reached, GoMaintain will automatically close the oldest index in the cluster to release memory resources used by the cluster. If warm nodes are present in the cluster, the index will automatically be moved to the warm nodes before the index is closed.

Force Merge Configuration

The force merge configuration combines index segments to improve search performance. In larger deployments, search performance can degrade over time due to a large number of segments. Force merge can alleviate this issue by optimizing older indices and reducing heap usage.

Enabling Force Merge will show these additional ForceMerge Settings:

Warm Tier Configuration

Beginning with LogRhythm SIEM version 7.19, a number of new configurable values have been added to optimize the Warm Tier search experience. These settings only apply to customers with Warm Tier Linux Data Indexer clusters, and can be configured independently for each DX cluster with different values.

Columbo Warm Tier Search Cycle Days

This value was designed to prevent over-subscription of the OpenSearch Heap Segment Terms memory by opening a controlled number of indexes during a given search cycle, then proceeding to paginate through the indexes to cover the duration of a search. 

For example, a company has a DX Cluster with 365 days of data, and today is January 1st, 2025.

• 30d Hot (Dec 2024)

• 60d Ultra-Warm (Oct/Nov 2024)

• 275d Warm-Closed (Jan-Sept 2024)

When an analyst runs a search for IP Address 10.10.10.10 for the month of April (30 days), if the "Columbo Warm Tier Search Cycle Days" option is configured for 10 days/indexes, the search cycles through three batches of opening, searching, and closing indexes before the search is completed. Following completion of the three batches, the Warm-Tier lock will be released.

When an analyst runs a search for IP Address 5.5.5.5 for February 10-16th, this only crosses seven days of indexes and, therefore, is completed in one batch.

• The default value in SIEM versions 7.19 and later is 20 days/indexes. Values over 30 days/indexes are considered experimental and may carry some risk of OOM condition on Warm Tier Indexers when combined with Ultra-Warm or Disabled Locking.

• The default value in versions prior to 7.19 is hard-coded at 5 days/indexes and cannot be modified.

Columbo Ultra-Warm Tier Open Days

Introduced in LogRhythm SIEM version 7.19, the Ultra-Warm Tier offers a much improved search experience over warm-closed while still taking advantage of cost-effective storage tiers for indexes which are not being actively written to. Ultra-Warm stores indexes on the Warm tier nodes, but leaves them open, taking advantage of the local heap memory available on each warm node for faster searching. 

Each open Ultra-Warm index consumes resources in Heap Memory and counts against max shards per node for the warm node on which the index resides. The more warm nodes present in your cluster, the more ultra-warm days you can safely open. 

Shard Maximums apply to Ultra-Warm Indexes/Nodes where each Warm node can hold a max of 2500 shards:

(Hot Node Count * 2 * Index Count) * 2 if warm-replica is enabled = Shard Count

For example, if a DX Cluster has 10 Hot + 1 Warm node without warm-replica (20 shards per index/2500 max shards in 1 warm node), the absolute maximym value for Ultra-Warm is 125 Days (values this high may negatively impact stability of your warm nodes and are not recommended).

In another example, if a DX Cluster has 6 Hot + 4 Warm nodes with warm-replica (24 shard per index/10,000 max shards across 4 warm nodes), the absolute max value for Ultra-Warm is 416 Days (values this high are very experimental and untested; use at your own discretion).

Columbo Warm Tier Locking

Introduced in LogRhythm SIEM version 7.19, this setting allows you to control the use of "Locking" in Warm Tier searching. In versions prior to LogRhythm SIEM 7.19, all warm tier search requests require an available lock to be run. These locks restricted warm tier to only a single concurrent search for each DX Cluster. With the introduction of Ultra-Warm tiers, the locking feature of warm-tier searching may not be applicable to some environments and can be disabled at the discretion of the user.

We recommend disabling locking in these scenarios:

• All Warm Data in the environment will be in Ultra-Warm (typically <60 days) and therefore always open.

• Searches against Warm-Closed data are exceptionally rare and would rarely be performed by multiple users at the same time.

We recommend enabling locking in these scenarios:

• Data in the Warm-Closed tier is frequently accessed many times per hour.

• Data in the Warm-Closed tier is accessed by many users concurrently, who may be searching overlapping time ranges.