Data Indexer Elasticsearch to OpenSearch Upgrades

LogRhythm version 7.26 introduced OpenSearch 2 as the Data Indexer backend, replacing the previous Elasticsearch 7 which was shipped between versions 7.18 and 7.25. OpenSearch 2 offers improved performance, better memory management, improved security, and a roadmap of long-term feature introduction in the Data Indexer.

Index/Data Compatibility

OpenSearch 2 is built on Lucene 9, which maintains backwards compatibility up to one major version (N-1), this means that OpenSearch 2 can read/write to indexes that were created in Lucene 8 (ES7) or Lucene 9 only. Indexes that were created in Lucene 7 (ES6) cannot be read by OpenSearch 2.

LogRhythm upgraded to Elasticsearch 7 in version 7.18; this means that indexes created in any LogRhythm version prior to 7.18 are not compatible with OpenSearch 2 in LogRhythm version 7.26+. This includes all index types (Hot, Warm, Ultra-Warm and Archive Restore (logsar-*). As part of the upgrade process to LogRhythm 7.26, the DX Upgrade Checker will validate if your cluster contains any incompatible indexes and remove them so the cluster can start after the upgrade is completed.

Example Scenarios

The following scenarios may help in understanding the upgrade process, when it is needed, and potential data loss situations.

If you are concerned your cluster may contain incompatible indexes, before upgrading to OpenSearch, you can confirm the version of each index and its compatibility with the scripts below.

Linux Script

curl -s "localhost:9200/_all/_settings/index.version.created?expand_wildcards=all" | \
jq -r 'to_entries[] | "\(.key) \(.value.settings.index.version.created)"' | \
while read -r index version; do
  case "$version" in
    6*) status="INCOMPATIBLE" ;;
    7*) status="COMPATIBLE" ;;
    *) status="UNKNOWN" ;;
  esac
  echo "$index $version $status"
done

Windows Script

# Query settings from local Elasticsearch
$uri = "http://localhost:9200/_all/_settings/index.version.created?expand_wildcards=all"
try {
    $settings = Invoke-RestMethod -Uri $uri -Method Get
} catch {
    Write-Error "Failed to connect to Elasticsearch: $_"
    return
}

# Iterate through each index and evaluate compatibility
foreach ($index in $settings.PSObject.Properties) {
    $indexName = $index.Name
    $version = $index.Value.settings.index.version.created

    # Determine compatibility based on leading digit of the index version
    $status = "UNKNOWN"
    if ($version) {
        if ($version.StartsWith("6")) {
            $status = "INCOMPATIBLE"
        } elseif ($version.StartsWith("7")) {
            $status = "COMPATIBLE"
        }
    }

    # Print the output in the same format
    Write-Host "$indexName $version $status"
}

Kibana Users

LogRhythm does not provide official support for Kibana or OpenSearch Dashboards; however, there are a significant number of deployments leveraging Kibana alongside LogRhythm 7.x. With the migration to OpenSearch 2, Kibana will no longer function. Customers using Kibana must migrate to OpenSearch Dashboards. For single-cluster deployments, scripts have been bundled with LogRhythm SIEM 7.26.0 to aid in the migration process; however, if you have a more complex Kibana deployment (such as multiple clusters), the migration is much more complex and not covered under LogRhythm Support.

If you wish to retain your existing Kibana custom configurations (dashboards, visualizations, etc.) and port them over to OpenSearch Dashboards, you must backup your Kibana Dashboards prior to the upgrade.

OpenSearch 2 introduces native security features which were not available in Elasticsearch 7, including enforced encryption and authentication in order to access data in the cluster. Your OpenSearch Dashboards configuration will need to take this into account. Customers with existing multi-cluster Kibana configurations are strongly encouraged to remove their Cross-Cluster Search (CCS) configurations and migrate to Multiple Data Sources (MDS) as this will offer a much simpler deployment.