Data Indexer Troubleshooting | LogRhythm SIEM

This guide addresses common concerns, issues, and commands needed to maintain your Data Indexer’s performance within the LogRhythm SIEM.

LogRhythm SIEM versions prior to 7.26 use Elasticsearch 7.10.2. Beginning with LogRhythm SIEM version 7.26, Elasticsearch was replaced by OpenSearch 2. Although both platforms perform the same core search and indexing functions, several administrative and troubleshooting workflows differ between versions.

The following table summarizes the key differences relevant to Data Indexer administration and support activities.

Area	Elasticsearch 7.x	OpenSearch 2.x
Protocol	HTTP	HTTPS
Authentication	Not Required	Required
Command Tool	curl	dx-curl
Security	Disabled by Default	Enabled by Default

Health Checks

For day-to-day monitoring, the Client Console/Web Console heartbeat, Grafana metrics, and service log files are typically sufficient to assess cluster health. However, when investigating issues or performing advanced troubleshooting, it may be necessary to interact directly with the Data Indexer backend.

LogRhythm strongly recommends that any backend commands which directly interact with Data Indexer services, indices, or cluster configuration are performed with a LogRhythm Support Engineer present. Executing these commands incorrectly may cause service interruptions, data loss, or cluster instability. LogRhythm/Exabeam cannot accept responsibility for issues resulting from unsupported or improperly executed backend operations.

Common Data Indexer Issues

Cluster Health During Upgrades

Before and after any upgrade, you should verify both the cluster health and the heartbeat status. The heartbeat timestamp should be current, and the status should remain clear (not Warning, Critical, or Red). This is particularly important during activities such as Elasticsearch to OpenSearch migrations, as index data may be moved, recovered, or reallocated during the process. Ensuring the cluster remains healthy helps confirm that data is available throughout the upgrade.

Green Cluster

At a high level, a green cluster is the desired state and indicates that all primary and replica shards are allocated and available. Cluster health should report 100% shard availability.

Yellow Cluster

A yellow cluster indicates that all primary shards are available, but one or more replica shards are unavailable. In a multi-node deployment, a temporary yellow state can be expected during maintenance, upgrades, or shard movement between nodes. However, a cluster that remains yellow for an extended period, or an active index (e.g. current date) that remains yellow, should be investigated and resolved. The cluster and shard allocation commands described later in this guide can help identify the cause.

Red Cluster

A red cluster indicates that one or more primary shards are unavailable. This means some data is inaccessible and requires immediate investigation. Temporary red states may occur during service restarts while shards come back online, but the cluster should recover automatically. A common issue is shard recovery becoming stuck at a specific percentage and never progressing. This is often a sign of a missing node, unavailable storage, or another allocation issue that should be remediated before proceeding.

Consul Connectivity and Time Synchronization

Consul-related issues are frequently caused by time synchronization problems between the Platform Manager and Data Indexer nodes. If systems are not synchronized to the same NTP source, cluster heartbeats can become misaligned, leading to Consul registration, service ownership, or service discovery issues.

This is particularly common when one node's clock is ahead of or behind another. If Consul issues are suspected, review the Consul and Data Indexer logs, verify NTP configuration, and ensure all participating systems are synchronized to the same reliable time source.

High Heap Utilization

High heap utilization is usually an indication that the cluster is under sustained load. Temporary increases during startup, recovery, upgrades, or shard reallocation are expected and typically resolve once cluster activity stabilizes.

However, persistent high heap usage (greater than 70%) is commonly a symptom of an undersized or oversubscribed environment. Common causes include insufficient disk performance, inadequate system resources, a high ingestion rate, or an imbalance between workload and available hardware.

During upgrades and recovery operations, LogRhythm guidance may recommend temporarily adjusting recovery settings to reduce cluster pressure while data is moved or recovered. If heap usage continues to remain elevated during normal operation, consider reviewing sizing recommendations and consulting Sales Engineering to determine whether the environment requires rescoping or additional resources.

OpenSearch Certificate Generation Failure

OpenSearch Security certificate generation failures prevent secure HTTPS communication between cluster nodes and clients. This typically presents as SSL/TLS errors in the OpenSearch logs, certificate validation failures, persistent 503 responses after startup, or OpenSearch failing to start altogether. In some cases, certificate files may exist but have become corrupted, incomplete, or otherwise invalid following an interrupted installation, upgrade, or manual modification.

Because certificate generation is skipped when existing certificate files are detected, simply re-running the installer may not resolve the issue. If certificate-related errors are suspected, remove the existing certificate files from the OpenSearch certificate directory and rerun the certificate generation tooling (Windows) or installer (NIX) to force a full certificate regeneration. In multi-node deployments, certificates should be removed from all affected nodes before regeneration to avoid mismatched trust relationships. After regeneration, verify that new certificate files have been created successfully and confirm cluster health and HTTPS connectivity before proceeding.

Service Relationships

The OpenSearch/Elasticsearch cluster provides data storage and search functionality. Transporter delivers data to the cluster, Columbo services user search requests, and API Gateway exposes backend services. In multi-node deployments, Carpenter, Bulldozer and GoMaintain operate under Consul leadership and are active on only a single node at a time.

Log Locations

In a healthy and correctly scaled deployment, the cluster should remain in a green/healthy state, with no persistent red or warning conditions. Log files should generally remain free of recurring errors, and services should not be repeatedly restarting during normal operation.

Bulldozer and Carpenter are key components responsible for maintaining the overall health and synchronization state of the Data Indexer cluster, effectively acting as part of the cluster's operational heartbeat within LRSIEM.

In multi-node deployments, an additional layer of complexity exists through Consul leader election and locking. Certain services operate in an active/passive model, running locally on all nodes but becoming active on only a single node at any given time based on the Consul lock owner.

Due to space limitations, the tables on this page may appear condensed with random line breaks. Clicking the “expand” button at the top-right of any table makes the table full-screen, alleviating some of the spacing issues.

Service	Purpose	What am I Looking for?	Linux Log Location	Multi-Node Behavior	Windows Log Location
OpenSearch	Search engine (LRSIEM 7.26+)	Cluster health, startup failures	`/var/log/OpenSearch/`	Runs on all nodes	`C:\Program Files\LogRhythm\Data Indexer\opensearch\logs`
Elasticsearch	Search engine (LRSIEM 7.25 and earlier)	Cluster health, startup failures	`/var/log/elasticsearch/`	Runs on all nodes	`C:\Program Files\LogRhythm\Data Indexer\elasticsearch\logs`
Dual Instance OpenSearch/Elasticsearch	Secondary search engine instance (-data portion for DX7500s+)	Cluster health, startup failures	`/var/log/elasticsearch/data/elastic` `/var/log/opensearch/data/opensearch/`	Present on dual-instance Linux setups only	`N/A`
API Gateway	API communication layer for Data Indexer services	Communication failures, backend connectivity issues	`/usr/local/logrhythm/LogRhythmAPIGateway/LogRhythmAPIGateway.log`	Runs on all nodes	`C:\Program Files\LogRhythm\LogRhythm Common\logs\LogRhythm API Gateway.log`
Service Registry (Consul)	Service discovery and cluster coordination	Service ownership, leader election, cluster membership	`/usr/local/logrhythm/LogRhythmServiceRegistry/LogRhythmServiceRegistry.log`	Runs on all nodes	`C:\Program Files\LogRhythm\LogRhythm Common\logs\LogRhythm Service Registry.log`
Transporter	Ships data to Data Indexer nodes and balances workload	Indexing issues	`/var/log/persistence/transporter.log`	Runs on all nodes	`C:\Program Files\LogRhythm\Data Indexer\logs\transporter.log`
Columbo	Processes searches, queries, and drilldowns	Search/Drilldown/Query issues	`/var/log/persistence/columbo.log`	Runs on all nodes	`C:\Program Files\LogRhythm\Data Indexer\logs\columbo.log`
Bulldozer	Updates EMDB with cluster health and node information	Heartbeat and health reporting issues	`/var/log/persistence/bulldozer.log`	Active on Consul leader node	`C:\Program Files\LogRhythm\Data Indexer\logs\bulldozer.log`
Carpenter	Synchronizes KB and deployment configuration to indices	KB sync, deployment sync issues	`/var/log/persistence/carpenter.log`	Active on Consul leader node	`C:\Program Files\LogRhythm\Data Indexer\logs\carpenter.log`
GoMaintain	Manages retention, disk space, and index lifecycle	Threshold regulation, hot/warm movement, disk management	`/var/log/persistence/gomaintain.log`	Active on Consul leader node	`C:\Program Files\LogRhythm\Data Indexer\logs\gomaintain.log`
Metrics	Grafana statistics collection	Data Indexer stats issues in Grafana	`/usr/local/logrhythm/LogRhythmMetrics/LogRhythmMetrics.log`	Runs on all nodes	`C:\Program Files\LogRhythm\LogRhythm Common\logs\LogRhythm Metrics Collection.log`

Interacting with the Data Indexer

For Linux operations, commands differ between the older ElasticSearch 7 (ES7) deployments (pre-LogRhythm SIEM version 7.26) and newer OpenSearch 2 (OS2) deployments (LogRhythm SIEM version 7.26 and later). With ES7, the API is accessible locally over HTTP and does not require authentication, so commands can query the cluster directly.

With OS2, security is enabled by default. API requests must use HTTPS and include the required authentication credentials. To simplify this, LogRhythm provides the pre-configured tools/dx-curl utility, which automatically handles authentication and certificate requirements.

As a result, the same health check operation is performed differently depending on the platform and version:

Platform	Function	Command
Elasticsearch	View cluster health	`curl localhost:9200/_cluster/health?pretty`
OpenSearch (Linux)	View cluster health	`/usr/local/logrhythm/tools/dx-curl.sh /_cluster/health?pretty`
OpenSearch (Windows)	View cluster health	`& "C:\Program Files\LogRhythm\Data Indexer\tools\dx-curl.ps1" /_cluster/health?pretty`

For ease of use when working with authenticated OpenSearch, you can create a dx-curl shortcut to avoid typing the full script path each time. Once configured, commands can be run using the simpler format dx-curl <endpoint>.

Create a Permanent dx-curl Shortcut on Linux

Create a symbolic link to the dx-curl.sh script: sudo ln -s /usr/local/logrhythm/tools/dx-curl.sh /usr/local/bin/dx-curl
Verify shortcut has been created: ls -l /usr/local/bin/dx-curl
Test shortcut: dx-curl /_cluster/health?pretty
The dx-curl command will now be available permanently without needing to specify the full script path or configure a shell alias.

Create a Permanent dx-curl Shortcut on Windows

Open your PowerShell profile: notepad $PROFILE
Add the following function: function dx-curl { & "C:\Program Files\LogRhythm\Data Indexer\tools\dx-curl.ps1" $args }
Save the profile and reload it: . $PROFILE
Verify the shortcut is available: dx-curl /_cluster/health?pretty
The shortcut will now be available automatically in future PowerShell sessions.

After completing these steps, OpenSearch commands can be run using the simpler dx-curl <endpoint> format instead of specifying the full script path each time.

Troubleshooting Workflow

When investigating Data Indexer issues, use the following approach:

Verify all expected services are running.
Check cluster health and shard availability.
Verify all expected cluster nodes are present.
Confirm Consul leadership ownership in multi-node deployments.
Review OpenSearch/Elasticsearch, Carpenter, Bulldozer and GoMaintain logs.
Check for unassigned shards or stalled shard recovery.
Review system resource utilization including CPU, memory, disk usage and heap consumption.
Escalate to LogRhythm Support before performing corrective actions.

Data Indexer Command Reference Sheet

A typical investigation should begin by confirming cluster health and shard availability, followed by verification of node membership and Consul leadership. If the cluster is not healthy, review unassigned shards, recovery progress, and relevant service logs before considering any corrective actions.

Administrative Commands (Read Only)

Function	Linux (Bash)	Example	Windows (PowerShell)
Check service status	`systemctl status <service>`	`systemctl status transporter`	`Get-Service <service>`
View all services	`systemctl list-units --type=service`	`systemctl list-units --type=service --state=running`	`Get-Service`
View service logs (see new, see latest x number)	`tail -f <location>` `tail -n <number> <location>`	`tail -f /var/log/opensearch/<clustername>.log` `tail -n 200 /var/log/opensearch/<clustername>.log`	`Get-Content "C:\Program Files\LogRhythm\LogRhythm Common\logs\LogRhythm API Gateway.log" -Tail 200`
Check data path mount location	`df -h <location>`	`df -h /home/logrhythm`	`$env:DXDATAPATH`

Health & Cluster Status (Read Only)

Function	What am I Looking For?	OpenSearch - Linux (Bash)	OpenSearch - Windows (PowerShell)	LEGACY Elasticsearch - Linux (Bash)
Cluster health refresh at interval (5 seconds)	Green cluster health and 100% of shards online	`watch -n 5 '/usr/local/logrhythm/tools/dx-curl.sh /_cluster/health?pretty'`	`while ($true) { Clear-Host; & "C:\Program Files\LogRhythm\Data Indexer\tools\dx-curl.ps1" /_cluster/health?pretty; Start-Sleep 5 }`	`watch -n 5 'curl -s localhost:9200/_cluster/health?pretty'`
Check all nodes	See which nodes are currently members of the cluster (any missing?)	`/usr/local/logrhythm/tools/dx-curl.sh /_cat/nodes?v`	`N/A`	`curl localhost:9200/_cat/nodes?v`
Check Consul leadership services	Identify which node owns services such as Bulldozer, GoMaintain, and Carpenter in a multi-node deployment	`sudo sh /usr/local/logrhythm/tools/consul_locks.sh`	`N/A`	`sudo sh /usr/local/logrhythm/tools/consul_locks.sh`
Recovery dashboard / cluster monitoring	Continuously monitor cluster health, nodes, heap usage, and non-green indices	`watch -n1 "echo 'Cluster health:' && /usr/local/logrhythm/tools/dx-curl.sh /_cluster/health?pretty && echo && echo 'Node load:' && /usr/local/logrhythm/tools/dx-curl.sh /_cat/nodes?v && echo && echo 'Elasticsearch Heap used:' && /usr/local/logrhythm/tools/dx-curl.sh '/_cluster/stats?human&pretty' \| grep -i heap && echo && echo 'Non-green indices:' && /usr/local/logrhythm/tools/dx-curl.sh /_cat/indices?v \| grep -v green"`	`while ($true) { Clear-Host; & "C:\Program Files\LogRhythm\Data Indexer\tools\dx-curl.ps1" /_cluster/health?pretty; Write-Host ''; & "C:\Program Files\LogRhythm\Data Indexer\tools\dx-curl.ps1" /_cat/nodes?v; Start-Sleep 5 }`	`watch -n1 "echo 'Cluster health:' && curl -s localhost:9200/_cluster/health?pretty && echo && echo 'Node load:' && curl -s localhost:9200/_cat/nodes?v && echo && echo 'Elasticsearch Heap used:' && curl -s 'localhost:9200/_cluster/stats?human&pretty' \| grep -i heap && echo && echo 'Non-green indices:' && curl -s localhost:9200/_cat/indices?v \| grep -v green"`

Index & Shard Operations (Read-Only)

Function	What am I Looking For?	OpenSearch - Linux (Bash)	OpenSearch - Windows (PowerShell)	LEGACY Elasticsearch - Linux (Bash)
List all indices	Typically you’d refine with a `\| grep <INDEX Name/State>`	`/usr/local/logrhythm/tools/dx-curl.sh /_cat/indices?v \| sort`	`& "C:\Program Files\LogRhythm\Data Indexer\tools\dx-curl.ps1" /_cat/indices?v`	`curl localhost:9200/_cat/indices?v`
List all non-green indices	Identify red/yellow indices	`/usr/local/logrhythm/tools/dx-curl.sh /_cat/indices?v \| grep -v green`	`(& "C:\Program Files\LogRhythm\Data Indexer\tools\dx-curl.ps1" /_cat/indices?v) \| Select-String -NotMatch "green"`	`curl localhost:9200/_cat/indices?v \| grep -v green`
Check unassigned shards	Display only shards that are not currently assigned to a node	`/usr/local/logrhythm/tools/dx-curl.sh /_cat/shards?v \| grep UNASSIGNED`	`(& "C:\Program Files\LogRhythm\Data Indexer\tools\dx-curl.ps1" /_cat/shards?v) \| Select-String "UNASSIGNED"`	`curl localhost:9200/_cat/shards?v \| grep UNASSIGNED`
Explain shard allocation	Determine why a shard is stuck	`/usr/local/logrhythm/tools/dx-curl.sh /_cluster/allocation/explain?pretty`	`& "C:\Program Files\LogRhythm\Data Indexer\tools\dx-curl.ps1" /_cluster/allocation/explain?pretty`	`curl localhost:9200/_cluster/allocation/explain?pretty`
Composition breakdown of hot/warm indices for multi-node tiers with warm nodes	Breakdown of hot/warm indices across multi node cluster environments. Hot indices = replica, open. Warm indices = no replica, closed (unless searched, then open/closed in batches). Ultra warm indices = no replica, open.	echo "Index Hot/Warm & Replica Summary:" && results=$(for i in $(sudo /usr/local/logrhythm/tools/dx-curl.sh '/_cat/indices?h=index' \| grep -i logs); do echo -n "$i: "; box=$(sudo /usr/local/logrhythm/tools/dx-curl.sh "/$i/_settings" 2>/dev/null \| grep -o '"box_type":"[^"]"' \| head -1 \| cut -d'"' -f4); replicas=$(sudo /usr/local/logrhythm/tools/dx-curl.sh "/$i/_settings" 2>/dev/null \| grep -o '"number_of_replicas":"[^"]"' \| head -1 \| cut -d'"' -f4); echo "box=$box replicas=$replicas"; done) && printf "%-10s %-10s %s\n" "TIER" "REPLICAS" "INDEX COUNT" && printf "%-10s %-10s %s\n" "----------" "----------" "-----------" && echo "$results" \| awk -F': ' '{print $2}' \| sort \| uniq -c \| awk '{box=$2; rep=$3; sub("box=","",box); sub("replicas=","",rep); printf "%-10s %-10s %s\n", box, rep, $1}' && echo && echo "Detailed results saved to /home/logrhythm/IndexComposition.txt" && echo "$results" > /home/logrhythm/IndexComposition.txt	`N/A`	echo "=== Index Hot/Warm & Replica Summary ===" && results=$(for i in $(curl -s "localhost:9200/_cat/indices?v" \| awk '{print $3}' \| grep -i logs); do echo -n "$i: "; box=$(curl -s "localhost:9200/$i/_settings" 2>/dev/null \| grep -o '"box_type":"[^"]"' \| head -1 \| cut -d'"' -f4); replicas=$(curl -s "localhost:9200/$i/_settings" 2>/dev/null \| grep -o '"number_of_replicas":"[^"]"' \| head -1 \| cut -d'"' -f4); echo "box=$box replicas=$replicas"; done) && echo "$results" \| awk -F': ' '{print $2}' \| sort \| uniq -c && echo && echo "Detailed results saved to IndexComposition.txt" && echo "$results" > IndexComposition.txt

Modification Commands (Consult LogRhythm Support First)

The commands in this section make live changes to cluster configuration, index allocation, shard placement, or data availability. Incorrect use may result in data becoming unavailable, shard relocation, reduced redundancy, or permanent data loss.

Function	What am I Looking For?	OpenSearch - Linux (Bash)	OpenSearch - Windows (PowerShell)	Legacy Elasticsearch - Linux (Bash)
Restart all non indexing services (not ElasticSearch / OpenSearch)	`systemctl restart <service>`	`sudo systemctl restart LogRhythmServiceRegistry LogRhythmAPIGateway transporter columbo gomaintain carpenter bulldozer LogRhythmMetricsCollection`	`Get-Service -DisplayName 'LogRhythm DX' \| Where-Object {$_.DisplayName -notlike 'Elastic' -and $_.DisplayName -notlike 'Open*'} \| Restart-Service`	`sudo systemctl restart LogRhythmServiceRegistry LogRhythmAPIGateway transporter columbo gomaintain carpenter bulldozer LogRhythmMetricsCollection`
Close / Open / Delete Index	Impacts live data availability.	Close: `/usr/local/logrhythm/tools/dx-curl.sh -XPOST /index_name/_close` Open: `/usr/local/logrhythm/tools/dx-curl.sh -XPOST /index_name/_open` Delete: `/usr/local/logrhythm/tools/dx-curl.sh -XDELETE /index_name`	Close: `& "C:\Program Files\LogRhythm\Data Indexer\tools\dx-curl.ps1" -XPOST /index_name/_close` Open: `& "C:\Program Files\LogRhythm\Data Indexer\tools\dx-curl.ps1" -XPOST /index_name/_open` Delete: `& "C:\Program Files\LogRhythm\Data Indexer\tools\dx-curl.ps1" -XDELETE /index_name`	Close: `curl -XPOST localhost:9200/index_name/_close` Open: `curl -XPOST localhost:9200/index_name/_open` Delete: `curl -XDELETE localhost:9200/index_name`
Change Hot/Warm Allocation	Move an index between Hot and Warm storage tiers.	`/usr/local/logrhythm/tools/dx-curl.sh -XPUT /<INDEXNAME>/_settings -H 'Content-Type: application/json' -d '{"index.routing.allocation.require.box_type":"hot"}'`	`& "C:\Program Files\LogRhythm\Data Indexer\tools\dx-curl.ps1" -XPUT /<INDEXNAME>/_settings -H "Content-Type: application/json" -d '{"index.routing.allocation.require.box_type":"hot"}'`	`curl -XPUT "localhost:9200/<INDEXNAME>/_settings" -H 'Content-Type: application/json' -d '{"index.routing.allocation.require.box_type":"hot"}'`
Alter Replica Count	Change replica count for an index. Setting `0` removes replicas. Setting `1` creates a single replica.	`/usr/local/logrhythm/tools/dx-curl.sh -XPUT /<INDEXNAME>/_settings -H 'Content-Type: application/json' -d '{"index":{"number_of_replicas":1}}'`	`& "C:\Program Files\LogRhythm\Data Indexer\tools\dx-curl.ps1" -XPUT /<INDEXNAME>/_settings -H "Content-Type: application/json" -d '{"index":{"number_of_replicas":1}}'`	`curl -XPUT "localhost:9200/<INDEXNAME>/_settings" -H 'Content-Type: application/json' -d '{"index":{"number_of_replicas":1}}'`
Retry Failed Shard Allocations	Retry allocation of shards previously marked as failed.	`/usr/local/logrhythm/tools/dx-curl.sh -XPOST /_cluster/reroute?retry_failed`	`& "C:\Program Files\LogRhythm\Data Indexer\tools\dx-curl.ps1" -XPOST /_cluster/reroute?retry_failed`	`curl -XPOST localhost:9200/_cluster/reroute?retry_failed`

Common Error Codes

The following HTTP response codes are commonly encountered during installation, upgrade, and troubleshooting activities. Understanding what they mean can help quickly identify whether a service is healthy, still starting, or experiencing a fault.

HTTP Code	Status	Common Meaning	Typical Action
200	OK	The request completed successfully and the service is responding as expected.	No action required.
401	Unauthorized	Authentication failed. Credentials may be incorrect, missing, or not yet fully initialized.	Verify credentials and allow additional time for service initialization if the cluster is still starting.
403	Forbidden	The request was authenticated but does not have permission to perform the requested action.	Remove Certificates folder from every node and rerun installer to recreate certificates.
404	Not Found	The requested endpoint, service, or configuration item could not be located.	Verify the URL, endpoint, service status, and Consul registration.
429	Too Many Requests	The target service is overloaded or receiving requests faster than it can process them.	Investigate workload, cluster utilization, and load balancing.
500	Internal Server Error	The service encountered an unexpected internal failure while processing the request.	Review application logs and verify dependent services are healthy.
502	Bad Gateway	A proxy or intermediary service cannot communicate with the backend service.	Verify network connectivity and backend service availability.
503	Service Unavailable	The service is online but temporarily unable to process requests. Common during startup or cluster initialization.	Allow startup to complete and monitor service health.