Search Tiers (Hot/Warm/Cold)

LogRhythm Enterprise Architecture offers three tiers of data storage/retrieval. Hot/Warm tiers exist within the Data Indexer cluster, where the Warm tier can be added on to any deployment with existing Linux-based Data Indexers. The Cold tier uses highly compressed archives, storing an original copy of the log. Warm tier search offers a more cost-effective storage profile while having an improved user experience from Cold (Secondlook).

Incoming Data Path

As log data is processed in the Data Processor, the logs are duplicated. One copy of the log is sent to the Transporter to be indexed/written to the DX Hot tier. After a configured retention period, if Warm tier is present, the logs will be moved from Hot to Warm. A second copy of the log is also written to the archive writer as it is processed. This means both the Hot Tier and the Archive are written to at the same time during processing.

Search Experience - Hot

Data in the Hot tier should be stored on an SSD, which provides a very fast retrieval time for users. When searching, Hot tier data retrieval times are typically in the seconds, but this can vary based on the criteria of the search, disk speeds, search result size, and load on the DX cluster.

When searching Hot tier within the Web Console, up to five search requests can be processed simultaneously (default, adjustable).

Search Experience - Warm

Data in the Warm tier is typically stored on a spinning HDD, which offers slower retrieval times but a much more cost-effective method of extending data retention. This means the search experience is expected to be slower, but offers a significant improvement over Secondlook retrieval times. Data in the Warm tier is stored in “closed” indexes which have to be opened upon request.

When searching Warm tier, only one search request can be processed simultaneously. Additional Warm tier search requests are queued until the Warm tier search requests in front of them complete or the search timeout is expired.

Warm tier uses a mechanism of opening/closing indexes when searches are performed. Indexes are opened/closed in chunks of five days; depending on the duration of the search, this can mean many rounds of opening/closing if the search is over an extended period of time.

1. User initiates a search of data from Warm Tier through Web Console or Client Console.

2. Colombo on the DX receives the request and detects the search request spans Warm tier. Since only one Warm tier search can run concurrently, Colombo initiates a lock.

3. Colombo closes all open indexes in the Warm tier which do not fall within the search duration dates.

4. Colombo search:

   1. Open five indexes,

   2. Search the five indexes,

   3. Return the results to the user from those five indexes,

   4. Closes the 5 indexes, and

   5. Repeat steps a through d until the duration of the search request is satisfied or the max results are hit, whichever comes first. For example, a search of 30-day Warm data will go through six cycles if it does not hit max results.

5. Colombo releases the Warm tier search lock.

Opening and Closing of indexes cause the cluster to temporarily go “red.” This is normal behavior when Warm tier searches are run. While the cluster is red, normal indexing and searching of the Hot tier still functions; however, this performance may be slightly degraded. Some cluster commands are not accepted (like creating a new index), but this should have no noticeable impact to the user.