Configure an Upgraded DR Deployment

Import the License File

This section describes how to import a license file and run the licensing wizard for each Data Processor in your deployment to ensure that the correct license has been assigned.

You must import a new license file and perform steps 5-7 on each Data Processor in your deployment.

For more information about licensing or the licensing wizard, see Assign LogRhythm Licenses.

1. Copy your LogRhythm SIEM license file to the Platform Manager or to a network location that is accessible from the Platform Manager.
2. Log on to a system where the current version of the LogRhythm Client Console is installed.
3. Import the LogRhythm SIEM license file:
   1. Start the Client Console and click Deployment Manager.
   2. On the File menu, click Import License File.
   3. Browse to and select your current-version LogRhythm SIEM license file, and then click Open.
      The License Import Warning is displayed: Importing a license synchronizes your licenses with the licenses in the file. It is important that you only import the latest license file issued you by LogRhythm. Importing an older license or one with a different master license ID may cause undesired results such as existing licensed components becoming unlicensed. Are you sure the selected file is your latest?
   4. Click Yes to continue importing the file.
   5. When the import is complete, click OK to close the Import Successful dialog box.
4. Click the Data Processors tab.
5. Double-click one of the Data Processors in the list.
   The Data Processor Properties dialog box appears.

6. Select a cluster from the Cluster Name list, and then click OK.

   

   Cluster information is sent out when applying configuration changes on the Data Indexer. For more information, see Configure the Data Indexer.

7. Review the current License Status and run the licensing wizard if necessary, and then click OK.
8. Repeat steps 5-7 on each Data Processor in your deployment.

Configure the Data Indexer

Configuring the Data Indexer for Windows and Linux has moved from the individual clusters, to the Configuration Manager on the Platform Manager. You can configure all data Indexers using the LogRhythm Configuration Manager installed on the Platform Manager.

• Cluster Name configuration is currently done through environment settings. Before configuring the Data Indexer on Windows, verify that the DX_ES_CLUSTER_NAME environment variable is set on both DR servers.
• LogRhythm Service Registry, LogRhythm API Gateway and LogRhythm Windows Authentication API Service must be running before opening LogRhythm Configuration Manager

• If you are configuring multiple data Indexers, all can be configured from the Primary PM as the configuration is centralized between servers.

  

  In an MSSP environment, DX Cluster names are visible to all Users of a Web Console, regardless of Entity segregation. For privacy reasons, avoid using cluster names that could be used to identify clients. Data and data privacy are still maintained; only the cluster name is visible

Do not attempt to modify consul configurations manually. If you have any issues, contact LogRhythm Customer Support.

To configure the Data Indexer:

1. Open the Configuration Manager from programs on the Platform Manager.

2. From the menu on the left, select the Data Indexers tab.

   Each installed Data Indexer has its own section that looks like this: 

   Data Indexer - Cluster Name: <ClusterName> Cluster Id: <ClusterID>

   

   The Cluster Name and Cluster ID come from the Environment variables, DX_ES_CLUSTER_NAME and DXCLUSTERID on each server. The Cluster Name can be modified in the Configuration Manager. If you change the Cluster Name, the name should be less than 50 characters long to ensure it displays properly in drop-down menus. The DXCLUSTERID is automatically set by the software and should not be modified.

3. Verify or update the following Data Indexer settings:

   

   Do not modify any settings from their defaults unless you fully understand their impact. Modifying a setting incorrectly can negatively impact Data Indexer function and performance.

   Setting	Default	Description
   Database User ID	LogRhythmNGLM	Username the DX services will use to connect to the EMDB database.  When in FIPS mode, Windows authentication is required (local or domain). When using a domain account, the Database Username must be in domain\username format.
   Database Password	<LogRhythm Default>	Password used by the DX services to connect to the EMDB database. It is highly recommended, and LogRhythm best practice, to change all MS SQL account passwords when setting up a deployment. After you change the LogRhythmNGLM password in Microsoft SQL Server Management Studio, you must set the Database Password to the same value. You should change the password in Microsoft SQL Server Management Studio first, then change it on the Data Indexer page.
   GoMaintain ForceMerge	Disabled	Enables/Disables maintenance Force Merging. This can be left at the default value.
   Integrated Security	Disabled	This should be enabled when FIPS is enabled on the operating system.

   

   Click Show or Hide in Advanced View to toggle the view for Advanced Settings.

   
   Advanced View Settings:

   Setting:	Default	Description
   Transporter Max Log Size (bytes)	1000000	Maximum log size in bytes that can be indexed. This can be left at the default value.
   Transporter Web Server Port	16000	Port that the Transporter service listens on. This can be left at the default value.
   Transporter Route Handler Timer (sec)	10	Indexing log batch timeout setting. This can be left at the default value.
   Elasticsearch Data Path	Windows: D:\LRIndexer\data Linux:/usr/local/logrhythm/db/data	Path where Data Indexer data will be stored. The path will be created if it does not already exist. Modifying this path after the Data Indexer installed will not move indices, they must be manually moved if the path is changed.
   GoMaintain TTL Logs (#indices)	-1	Number of indices kept by the DX. This should be left at the default value.
   GoMaintain IndexManage Elasticsearch Sample Interval (sec)	10	Number of seconds between resource usage samples. This can be left at the default value.
   GoMaintain Elasticsearch Samples (#Samples)	60	Total number of samples taken, before GoMaintain decides to take action, when resource HWMs are reached.
   GoMaintain IndexManager Disk HWM (%diskutil)	80	Maximum percentage of the disk for the Drive where the data path is configured. This can be left at the default value.
   GoMaintain IndexManage Elasticsearch Heap HWM (%esheap)	85	Maximum % Heap used percentage before GoMaintain closes an index to release resources. This can be left at the default value.
   Carpenter SQL Paging Size (#records)	10000	Number of records to pull from EMDB at one time when syncing EMDB indices. This can be left at the default value.
   Carpenter EMDB Sync Interval (#minutes)	5	Interval of how often Carpenter service will sync EMDB indices. This can be left at the default value.
   Enable Warm Replicas	Disabled	Turn replicas on for Warm Indices. This setting will only affect Linux Data Indexer clusters that contain warm nodes. This can be left at the default value.

4. Click Submit.

Automatic Maintenance

Automatic maintenance is governed by several of the above settings by the GoMaintain service. On startup, GoMaintain will continuously take samples from Elasticsearch stats, including disk and heap utilization for the configured time frame. 

GoMaintain will automatically perform maintenance when High Water Mark settings are reached. Samples are taken over a period of time and analyzed before GoMaintain will take action on an index. This will depend on the Sample Interval and #Sample settings. By default, this is 60 samples, 1 every 10 seconds for a total of 10 minutes. If it is determined during that sample period that a High Water Mark setting was reached for an extended period of time, indices will be closed, deleted, or moved to warm nodes depending on the data indexer configuration. After an action is taken and completed, the sample period will begin again.

The DX monitors Elasticsearch memory and DX storage capacity. GoMaintain tracks heap pressure on the nodes. If the pressure constantly crosses the threshold, GoMaintain decreases the number of days of indices by closing the index. Closing the index removes the resource needs of managing that data and relieves the heap pressure on Elasticsearch. GoMaintain continues to close days until the memory is under the warning threshold, and continues to delete days based on the default disk utilization setting of 80%.

Logging of configuration and results for force merge can be found in C:\Program Files\LogRhythm\DataIndexer\logs\GoMaintain.log.

GoMaintain TTL Logs (#Indices)

The default configuration value is -1. This value monitors the systems resources and automatically manages the time-to-live (TTL). You can configure a lower TTL by changing this number. If this number is no longer achievable, the Data Indexer sends a diagnostic warning and starts closing the indices. Indices that have been closed by GoMaintain are not actively searchable after 7.9.x, but are maintained for reference purposes. 

To show closed indices, run a curl command such as:

curl -s -XGET 'http://localhost:9200/_cat/indices?h=status,index' | awk '$1 == "close" {print $2}'

To show both open and closed indices, open a browser to http://localhost:9200/_cat/indices?v.

Indices can be reopened with the following query, as long as you have enough heap memory and disk space to support this index. If you do not, it immediately closes again.

curl -XPOST 'localhost:9200/<index>/_open?pretty'

After you open the index in this way, you can investigate the data in either the Web Console or Client Console.

Disk Utilization Limit

• IndexManager Disk HWM (%diskUtil) Indicates the percentage of disk utilization that triggers maintenance. The default is 80, which means that maintenance starts when the Elasticsearch data disk is 80% full.

• If Warm nodes are present, the disk utilization for combined Hot and Warm nodes will be tracked separately.

  

  The value for %diskUtil should not be set higher than 80. This can have an impact on the ability of Elasticsearch to store replica shards for the purpose of failover.

If Warm nodes are present, the oldest index will be moved to the Warm node(s) if the Disk HWM is reached.

Maintenance is applied to the active repository, as well as archive repositories created by Second Look. When the Disk Usage Limit is reached, active logs are trimmed when “max indices” is reached. At this point, GoMaintain deletes completed restored repositories starting with the oldest date.

The default settings prioritize restored repositories above the active log repository. Restored archived logs are maintained while sacrificing active logs. If you want to keep your active logs and delete archives for space, set your min indices equal to your max indices. This forces the maintenance process to delete restored repositories first.

Heap Utilization Limit

• IndexManager Heap HWM (%esheap) Indicates the percentage of Elasticsearch (java) heap utilization that triggers maintenance. The default is 85, which means that maintenance starts when the Elasticsearch heap utilization reaches 85%.

  

  The value for %esheap should not be set higher than 85. This can have an impact on the ability of Elasticsearch searches and indexing and can degrade overall Elasticsearch performance.

• If the Heap HWM is reached, GoMaintain will automatically close the oldest index in the cluster to release memory resources used by the cluster. If warm nodes are present in the cluster, the index will automatically be moved to the warm nodes before the index is closed.

  

  Closed Indices on Hot nodes cannot be searched and will remain in a closed state on the data indexer until the Utilization Limit is reached.

Force Merge Configuration

Do not modify any of the configuration options under Force Merge Config without the assistance of LogRhythm Support or Professional Services.

The force merge configuration combines index segments to improve search performance. In larger deployments, search performance can degrade over time due to a large number of segments. Force merge can alleviate this issue by optimizing older indices and reducing heap usage.

Enabling Force Merge will show these additional ForceMerge Settings: