Perform Post-Upgrade Procedures on the Upgraded LogRhythm Deployment

After you have upgraded all your appliances, you can perform a few configuration tasks and restart the LogRhythm solution.

Reconfigure CloudAI

Configure the LogRhythm Data Indexer

Configuring the Data Indexer for Windows and Linux has moved from the individual clusters, to the Configuration Manager on the Platform Manager. You can configure all data Indexers using the LogRhythm Configuration Manager installed on the Platform Manager.

• Cluster Name configuration is currently done through environment settings. Before configuring the Data Indexer on Windows, verify that the DX_OS_CLUSTER_NAME environment variable is set on both DR servers.

• LogRhythm Service Registry, LogRhythm API Gateway and LogRhythm Windows Authentication API Service must be running before opening LogRhythm Configuration Manager

• If you are configuring multiple data Indexers, all can be configured from the Primary PM as the configuration is centralized between servers.

To configure the Data Indexer:

1. Open the Configuration Manager from programs on the Platform Manager.

2. From the menu on the left, select the Data Indexers tab.

   Each installed Data Indexer has its own section that looks like this: 

   Data Indexer - Cluster Name: <ClusterName> Cluster Id: <ClusterID>

3. Verify or update the following Data Indexer settings:

Advanced View Settings:

4. Click Submit.

Automatic Maintenance

Automatic maintenance is governed by several of the above settings by the GoMaintain service. On startup, GoMaintain will continuously take samples from OpenSearch stats, including disk and heap utilization for the configured time frame. 

GoMaintain will automatically perform maintenance when High Water Mark settings are reached. Samples are taken over a period of time and analyzed before GoMaintain will take action on an index. This will depend on the Sample Interval and #Sample settings. By default, this is 60 samples, 1 every 10 seconds for a total of 10 minutes. If it is determined during that sample period that a High Water Mark setting was reached for an extended period of time, indices will be closed, deleted, or moved to warm nodes depending on the data indexer configuration. After an action is taken and completed, the sample period will begin again.

The DX monitors OpenSearch memory and DX storage capacity. GoMaintain tracks heap pressure on the nodes. If the pressure constantly crosses the threshold, GoMaintain decreases the number of days of indices by closing the index. Closing the index removes the resource needs of managing that data and relieves the heap pressure on OpenSearch. GoMaintain continues to close days until the memory is under the warning threshold, and continues to delete days based on the disk utilization setting.

Logging of configuration and results for force merge can be found in C:\Program Files\LogRhythm\DataIndexer\logs\GoMaintain.log.

GoMaintain TTL Logs (#Indices)

The default configuration is -1. This value monitors the systems resources (Heap and Disk) and automatically manages the time-to-live (TTL). You can configure a lower/static TTL by changing this number. If this number is no longer achievable due to heap consumption, the DX sends a diagnostic warning and starts closing the indices or moving them to Warm nodes if present.

Disk Utilization Limit

The disk utilization limit indicates the percentage of disk utilization that triggers maintenance. The recommended value depends on the type of disks used in your Hot DX nodes: 90% for SSD and 80% for HDD. This value triggers when maintenance starts based on disk consumption of the smallest Hot node in the cluster. Maintenance for GoMaintain will either delete the oldest index or move it from Hot to Warm tier if Warm tier is present in the cluster. The value for Disk Utilization Limit should not be set higher than 90. This value can have an impact on the ability of OpenSearch to store replica shards for the purpose of failover.

• IndexManager Disk HWM (%diskUtil) Indicates the percentage of disk utilization that triggers maintenance.

• If Warm nodes are present, the disk utilization for combined Hot and Warm nodes will be tracked separately.

If Warm nodes are present, the oldest index will be moved to the Warm node(s) if the Disk HWM is reached.

Maintenance is applied to the active repository, as well as archive repositories created by Second Look. When the Disk Usage Limit is reached, active logs are trimmed when “max indices” is reached. At this point, GoMaintain deletes completed restored repositories starting with the oldest date.

The default settings prioritize restored repositories above the active log repository. Restored archived logs are maintained while sacrificing active logs. If you want to keep your active logs and delete archives for space, set your min indices equal to your max indices. This forces the maintenance process to delete restored repositories first.

Heap Utilization Limit

• IndexManager Heap HWM (%esheap) Indicates the percentage of OpenSearch (java) heap utilization that triggers maintenance. The default is 85, which means that maintenance starts when the OpenSearch heap utilization reaches 85%.

• If the Heap HWM is reached, GoMaintain will automatically close the oldest index in the cluster to release memory resources used by the cluster. If warm nodes are present in the cluster, the index will automatically be moved to the warm nodes before the index is closed.

Force Merge Configuration

The force merge configuration combines index segments to improve search performance. In larger deployments, search performance can degrade over time due to a large number of segments. Force merge can alleviate this issue by optimizing older indices and reducing heap usage.

Enabling Force Merge will show these additional ForceMerge Settings:

Warm Tier Configuration

Beginning with LogRhythm SIEM version 7.19, a number of new configurable values have been added to optimize the Warm Tier search experience. These settings only apply to customers with Warm Tier Linux Data Indexer clusters, and can be configured independently for each DX cluster with different values.

Columbo Warm Tier Search Cycle Days

This value was designed to prevent over-subscription of the OpenSearch Heap Segment Terms memory by opening a controlled number of indexes during a given search cycle, then proceeding to paginate through the indexes to cover the duration of a search. 

For example, a company has a DX Cluster with 365 days of data, and today is January 1st, 2025.

• 30d Hot (Dec 2024)

• 60d Ultra-Warm (Oct/Nov 2024)

• 275d Warm-Closed (Jan-Sept 2024)

When an analyst runs a search for IP Address 10.10.10.10 for the month of April (30 days), if the "Columbo Warm Tier Search Cycle Days" option is configured for 10 days/indexes, the search cycles through three batches of opening, searching, and closing indexes before the search is completed. Following completion of the three batches, the Warm-Tier lock will be released.

When an analyst runs a search for IP Address 5.5.5.5 for February 10-16th, this only crosses seven days of indexes and, therefore, is completed in one batch.

• The default value in SIEM versions 7.19 and later is 20 days/indexes. Values over 30 days/indexes are considered experimental and may carry some risk of OOM condition on Warm Tier Indexers when combined with Ultra-Warm or Disabled Locking.

• The default value in versions prior to 7.19 is hard-coded at 5 days/indexes and cannot be modified.

Columbo Ultra-Warm Tier Open Days

Introduced in LogRhythm SIEM version 7.19, the Ultra-Warm Tier offers a much improved search experience over warm-closed while still taking advantage of cost-effective storage tiers for indexes which are not being actively written to. Ultra-Warm stores indexes on the Warm tier nodes, but leaves them open, taking advantage of the local heap memory available on each warm node for faster searching. 

Each open Ultra-Warm index consumes resources in Heap Memory and counts against max shards per node for the warm node on which the index resides. The more warm nodes present in your cluster, the more ultra-warm days you can safely open. 

Shard Maximums apply to Ultra-Warm Indexes/Nodes where each Warm node can hold a max of 2500 shards:

(Hot Node Count * 2 * Index Count) * 2 if warm-replica is enabled = Shard Count

For example, if a DX Cluster has 10 Hot + 1 Warm node without warm-replica (20 shards per index/2500 max shards in 1 warm node), the absolute maximym value for Ultra-Warm is 125 Days (values this high may negatively impact stability of your warm nodes and are not recommended).

In another example, if a DX Cluster has 6 Hot + 4 Warm nodes with warm-replica (24 shard per index/10,000 max shards across 4 warm nodes), the absolute max value for Ultra-Warm is 416 Days (values this high are very experimental and untested; use at your own discretion).

Columbo Warm Tier Locking

Introduced in LogRhythm SIEM version 7.19, this setting allows you to control the use of "Locking" in Warm Tier searching. In versions prior to LogRhythm SIEM 7.19, all warm tier search requests require an available lock to be run. These locks restricted warm tier to only a single concurrent search for each DX Cluster. With the introduction of Ultra-Warm tiers, the locking feature of warm-tier searching may not be applicable to some environments and can be disabled at the discretion of the user.

We recommend disabling locking in these scenarios:

• All Warm Data in the environment will be in Ultra-Warm (typically <60 days) and therefore always open.

• Searches against Warm-Closed data are exceptionally rare and would rarely be performed by multiple users at the same time.

We recommend enabling locking in these scenarios:

• Data in the Warm-Closed tier is frequently accessed many times per hour.

• Data in the Warm-Closed tier is accessed by many users concurrently, who may be searching overlapping time ranges.

Import the License File

This section describes how to import a license file and run the licensing wizard for each Data Processor in your deployment to ensure that the correct license has been assigned.

1. Copy your LogRhythm SIEM license file to the Platform Manager or to a network location that is accessible from the Platform Manager.

2. Log on to a system where the current version of the LogRhythm Client Console is installed.

3. Start the Client Console and click Deployment Manager.

4. On the File menu, click Import License File.

5. Browse to and select your current-version LogRhythm SIEM license file, and then click Open.
   The License Import Warning is displayed.

6. Click Yes to continue importing the file.

7. When the import is complete, click OK to close the Import Successful dialog box.

8. Click the Data Processors tab.

9. Double-click one of the Data Processors in the list.
   The Data Processor Properties dialog box appears.

10. Select a cluster from the Cluster Name list, and then click OK.
    Cluster information is sent out when applying configuration changes on the Data Indexer. For more information, see Configure the Data Indexer.

11. Review the current License Status and run the licensing wizard if necessary, and then click OK.

12. Repeat steps 9 through 11 on each Data Processor in your deployment.

Start the LogRhythm Solution

You need to do the following on each of the appliances in your deployment.

1. Log on to the appliance as an administrator.

2. On the Windows Data Indexer, run C:\Program Files\LogRhythm\Data Indexer\tools\start-allservices.bat as an administrator.

3. Open the Services Control Panel/App.

4. For any LogRhythm services that are not running, right-click the service name, and then click Start. These services include:

   • LogRhythm core services (AI Engine services, Alarming and Response Manager, Job Manager, Mediator Server Service, System Monitor)

   • All LogRhythm Web Console services

   • Any other “LogRhythm” services

5. Press F5 to refresh the list and verify that the services are started.

6. On Linux Data Indexers, run /usr/local/logrhythm/tools/start-all-services-linux.sh using sudo.

Upgrade and Start Other Agents

Agents on other collectors and aggregators can be upgraded at any time, although we recommend upgrading them now.

Windows

Users who attempt to upgrade the 32-bit System Monitor on a system having MS KB2918614 applied may encounter a software restriction policy error. In this case, the existing System Monitor should be uninstalled before attempting the upgrade.

To upgrade System Monitor Agent, do the following:

1. Log on with the Administrator account, or an account having administrative privileges, to the system where the System Monitor Agent is installed.

2. Before removing System Monitor, verify the account used for the System Monitor Service. In Windows Services console, right-click and view Properties, and click the Log On tab. If the service does not use Local System account, you will need the password to that account when installing the Agent, or you need a new account and password.

3. To open Windows Services, click Start, Administrative Tools, and Services.

4. Stop the service called LogRhythm System Monitor service.

5. Run the installer for the 32-bit or 64-bit System Monitor Agent, LRSystemMonitor_#.#.#.#.exe.

6. If the system does not have the Microsoft Visual C++ 2010 Redistributable Package installed, click Install.

7. Follow the instructions in the Install Wizard.

8. If prompted, accept the license agreement.

9. Choose the default installation path, and then click Next.

10. To determine if you have sufficient space for installation, click Space and then click OK.

11. (Optional.) Install the Realtime FIM driver.

12. On the Install Wizard Completed screen, clear the Launch System Monitor Configuration Manager check box.

13. If your LogRhythm Windows System Monitor Agent service uses Windows accounts, open Windows Services Control Panel.

14. Click the Log On tab and add the service account and password in the service properties.

15. To start the Agent, click Start, Administrative Tools, and Services. Right-click the agent and select Start.

16. Repeat these steps for other Windows System Monitor Agents in your deployment.

(Optional.) *NIX

Only Linux 2.4 & 2.6 can be upgraded directly. Other *NIX agents must be uninstalled and reinstalled.

1. Copy the files from the installer package to the *NIX system.

2. Follow the instructions in the scsm_<operating_system>.txt file to uninstall the old version.

3. Decompress the file with the .tar extension, using tar xf scsm_<operating_system>.tar.

4. Follow the instructions in the scsm_<operating_system>.txt file to install the new *NIX System Monitor Agent.

5. Start the *NIX System Monitor Agent according to the instructions in the scsm_<operating_ system>.txt file.

6. Repeat for all *NIX Agents in your deployment.

Configure or Verify Communication Ports

LogRhythm installers should open the TCP ports required for component communications. Additional configuration may be required, as described in this section. For more information on ports, see the Networking and Communication topic in the LogRhythm SIEM Documentation.

Configure Access for Remote Consoles

Users should access their LogRhythm deployment using a Client Console that is installed on their local workstation or through Citrix/Terminal Services (that is, not via the Client Console that is installed on the XM or Event Manager/Platform Manager). For this reason, some configuration to allow remote access may be required after upgrading to the latest version of LogRhythm SIEM.

Verify Ports on the Linux Data Indexer

To verify which ports are listening for incoming traffic on a Linux Indexer node, log on to the Indexer node as logrhythm and run the following command:

sudo firewall-cmd --permanent --zone=public --list-all

This lists all the public ports opened for DX: 

• 8501/tcp

• 8300/udp

• 8301/udp

• 8300/tcp

• 8301/tcp

If you need to open any incoming ports on the Linux Indexer, do the following:

1. Log on to the Indexer node as logrhythm and run the following commands:

   sudo firewall-cmd --zone=public --add-port=port/tcp --permanent
   sudo firewall-cmd –-reload
   

2. Repeat the steps above on each Linux Data Indexer.

Verify Ports on the Windows Data Indexer or the Data Processor

To verify allowed ports on a Windows server host:

1. Log on to the Windows server as an administrator.

2. Open a command prompt and run the following command:

   netsh firewall show state
   

   Ports that are currently open on all interfaces are displayed below the firewall status.

If you need to allow any ports on a Windows server host:

1. Log on to the Windows server as an administrator.

2. Open a command prompt and run the following command:

   netsh advfirewall firewall add rule name="rule name" dir=in action=allow protocol=TCP localport=port
   

Add Realtime Antivirus Exclusions for LogRhythm

Endpoint Security software, including Anti-Virus, Anti-Malware, and EDR/EPP solutions, can have a major impact on Installation, Upgrade, and ongoing Operations of any high-performance application, which includes the LogRhythm platform. LogRhythm provides a recommended list of paths to exclude from Realtime Scanning as a best practice to reduce the performance/stability impacts that can negatively affect the software. In some cases, there may be features specific to your Endpoint Security vendor, such as Heuristic detections, which may be required to be disabled due to vendor incompatibility. The directories below should be considered a minimum list of exclusion paths and additional paths may be required in some situations.

XM

If you have an XM appliance, apply the exclusions specified for all services below 

PM Services

• D:\LogRhythm\*.mdf

• L:\LogRhythm\*.ldf

• T:\Temp\*.mdf

• T:\Temp\*.ldf

• C:\Program Files\LogRhythm\LogRhythm Common\*

• C:\Program Files\LogRhythm\LogRhythm Alarming and Response Manager\*

• C:\Program Files\LogRhythm\LogRhythm Job Manager\*

• C:\Program Files\LogRhythm\LogRhythm Metrics\*

• If the Threat Intelligence Service (TIS) is installed:

  • C:\Program Files\LogRhythm\LogRhythm Threat Intelligence Service\*

DP Services (Windows)

• S:\LogRhythmArchives\Active\*.lua

• D:\LogRhythmArchives\Inactive\*.lca

• C:\Program Files\LogRhythm\LogRhythm Common\*

• S:\LogRhythm\LogRhythm Mediator Server\state\*

DX Services (Linux)

• /usr/local/logrhythm/db/opensearch/data

DX Services (Windows XM)

• All files in the directories and sub-directories of the paths stored in the environment variables %DXPATH% and %DXDATAPATH%. (Defaults below)

  • D:\LRIndexer

  • C:\Program Files\LogRhythm\Data Indexer

AIE Services

• S:\Program Files\LogRhythm\LogRhythm AI Engine\data\*.

• S:\Program Files\LogRhythm\LogRhythm AI Engine\state\*

• C:\Program Files\LogRhythm\LogRhythm System Monitor\state\*.pos

• C:\Program Files\LogRhythm\LogRhythm Common\*

• C:\Program Files\LogRhythm\LogRhythm AI Engine\*

• S:\Program Files\LogRhythm\LogRhythm Archive Engine\*

System Monitor Agent (Windows)

• C:\Program Files\LogRhythm\LogRhythm System Monitor\state\*.bin

• C:\Program Files\LogRhythm\LogRhythm System Monitor\state\*.pos

• C:\Program Files\LogRhythm\LogRhythm System Monitor\state\*.suspense

System Monitor Agent (Legacy Linux)

• /opt/logrhythm/scsm/state/*.pos

• /opt/logrhythm/scsm/state/*.suspense

Web Console Services

• C:\Program Files\LogRhythm\LogRhythm Web Services\*

• S:\tmp\indices\* (also often S:\LogRhythm\webindices\*)

• C:\Program Files\LogRhythm\LogRhythm Common\*

Secondlook API

• C:\Windows\system32\config\systemprofile\AppData\Roaming\LogRhythm\temp\

High Availability Deployments

• C:\lk\* directory (or whichever folder LifeKeeper is installed in)

• C:\Program Files (x86)\SIOS\DataKeeper> directory (or whichever folder DataKeeper is installed in)

• C:\Program Files (x86)\SIOS\DataKeeper\Bitmaps) (or whichever folder the bitmap file is stored in)

• HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Class\{71A27CDD-812A-11D0- BEC7-08002BE2092F}

• Registry keys used by SIOS are available at the following link: SIOS DataKeeper Administration

Verify Web Console Processes

The installer automatically starts the services and processes needed to run the Web Console. However, you should ensure that these processes are running by doing the following:

1. Go to Services on your machines.

2. Verify that the following services have started:

   • LogRhythm API Gateway

   • LogRhythm Authentication API

   • LogRhythm Case API

   • LogRhythm Service Registry

   • LogRhythm Threat Intelligence API

   • LogRhythm Web Console API

   • LogRhythm Web Console UI

   • LogRhythm Web Indexer

   • LogRhythm Web Services Host API

3. Go to Task Manager on your machine.

4. Verify that the following services have started:

   • java.exe (one instance)

   • LogRhythm.Web.Services.ServicesHost.exe

   • LogRhythmAPIGateway.exe

   • LogRhythmAuthenticationAPI.exe

   • LogRhythmCaseAPI.exe

   • LogRhythmServiceRegistry.exe

   • LogRhythmThreatIntelligence.exe

   • lr-threat-intelligence-api.exe (32 bit)

   • LogRhythmWebConsoleAPI.exe

   • LogRhythmWebConsoleUI.exe

   • LogRhythmWebIndexer.exe

   • LogRhythmWebServicesHostAPI.exe

   • nginx.exe *32 (a minimum of two instances)

   • node.exe (four instances)

   • procman.exe (eight instances)

   • NSSM Service Manager

Update Dashboards

In 7.6, LogRhythm updated the default dashboards in the Web console for new deployments. The Executive, IT Operations, and Security Analyst dashboards, along with the default Analyze pages, have been updated to include new widgets and added colors.

Because some dashboards have been customized in user environments, existing LogRhythm users cannot access this feature automatically. To import these dashboards in an existing deployment, choose one of the following options:

1. Run any of the deploy scripts to update the existing dashboard with the new one. These scripts include:

   • Web system update default Analyze Dashboard workscript.sql

   • Web system update default Executive Dashboard workscript.sql

   • Web system update default Security Analyst Dashboard Dashboard workscript.sql

   • Web system update default IT Operations Dashboard workscript.sql

2. Import the dashboards as new:

   • Analyze Dashboard.wdlt

   • Executive Dashboard.wdlt

   • Security Analyst Dashboard.wdlt

   • IT Operations Dashboard.wdlt

For instructions on how to import dashboards, see Import and Export Dashboards.

To download the dashboard files, visit the "Documentation & Downloads" section of the LogRhythm Community.

Remove Generic Web Console UI Entry from the Configuration Manager

Web Console users who upgraded from LogRhythm 7.2 or 7.3 will have a section in the LogRhythm Configuration Manager for Web Console UI, without a hostname designation to identify the Web Console server. This section should be removed from the Configuration Manager.

1. Log in to any server where the Web Console is installed as a user with administrative privileges.

2. Open a PowerShell window and run the following command:

   Invoke-RestMethod -Method Delete http://localhost:8500/v1/kv/services/lr-web-console-ui/CONFIG_OPTIONS

3. Restart the LogRhythm Web Console UI service on all Web Console servers.

Set KB Downloads to Automatic

LogRhythm recommends that you configure automatic download and synchronization of your KB. If you did not already set this option when synchronizing the latest KB, do the following:

1. On the Tools menu, click Knowledge, and then click Knowledge Base Manager.
   The Knowledge Base Manager appears.

2. Click Synchronization Settings.

3. Select the following check boxes:

   1. Enable Automatic Knowledge Base Download.

   2. Enable Automatic Knowledge Base Core Synchronization.

   3. Enable Automatic Knowledge Base Module Synchronization.

4. If a proxy is required, click the Proxy Settings tab.

   1. Enter the Proxy Server Address and Proxy Server Port.

   2. If the proxy server requires authentication, select that check box and provide a valid User Name, Password, and Domain.

5. Click OK.

6. To force an immediate download, click Check for Knowledge Base Updates.

Remove FIM State File

In a previous release, if the name of any FIM policies contained one or more colon ":" characters, *NIX Agents were unable to load state files. This issue has been fixed. To eliminate any further error messages following the upgrade, delete the existing Agent state file (../state/.filemon.log).