Search Tiers (Hot/Ultra-Warm/Warm/Cold)
LogRhythm Enterprise Architecture offers four tiers of data storage/retrieval. Hot/Ultra-Warm/Warm tiers exist within the Data Indexer cluster, where the Ultra-Warm/Warm tier can be added on to any deployment with existing Linux-based Data Indexers. The Cold tier uses highly compressed archives, storing an original copy of the log. The Ultra-Warm tier offers a fast search experience on indexes which are not actively written to, but open for, search on a more cost effective tier of storage. Warm tier offers extended TTL (3m-1yrs) on cost-effective storage while having an improved user experience from Cold (Secondlook). Each LogRhythm deployment may be designed to different sizes for each tier depending on the architecture, logs, volume, etc.
Tier | Index Status | Typical Search Speed | Ideal TTL | Compression | Disk Type |
|---|---|---|---|---|---|
Hot | Open - Writing | Seconds | 30-90 Days | DEFLATE ~1:0.86 | SSD |
Ultra Warm | Open - Read Only | Seconds | 60-180 Days | DEFLATE ~1:0.86 | HDD 7k |
Warm | Closed | Minutes | 60-365 Days | DEFLATE ~1:0.86 | HDD 7k |
Cold | N/A | Hours | Unlimited | GZIP ~10:1 | HDD 7k |
Incoming Data Path
As log data is processed in the Data Processor, the logs are duplicated. One copy of the log is sent to the Transporter to be indexed/written to the DX Hot tier. After a configured retention period, if Warm tier is present, the logs will be moved from Hot to Warm. A second copy of the log is also written to the archive writer as it is processed. This means both the Hot Tier and the Archive are written to at the same time during processing.
Search Experience - Hot
Data in the Hot tier should be stored on an SSD, which provides a very fast retrieval time for users. When searching, Hot tier data retrieval times are typically in the seconds, but this can vary based on the criteria of the search, disk speeds, search result size, and load on the DX cluster.
When searching the Hot tier within the Web Console, up to five search requests can be processed simultaneously (default, adjustable).
Search Experience - Ultra-Warm
Data in the Ultra-Warm tier is typically stored on a spinning HDD, which offers slower retrieval times but a much more cost-effective method of extending data retention than Hot. Data in the Ultra-Warm tier is stored in “open” indexes which have been closed for writing but are open for fast search at any time.
When searching the Warm (ultra or closed) tier, only one search request can be processed simultaneously. Additional Warm tier search requests are queued until the Warm tier search requests in front of them complete or the search timeout is expired. As of LogRhythm version 7.19, this value is configurable and can be disabled. For customers where their entire Warm tier is Ultra-Warm (typically 90 days or less), it is recommend to disable the locking.
Search Experience - Warm
Data in the Warm tier is typically stored on a spinning HDD using the same nodes as Ultra-Warm. This means the search experience is expected to be slower, but offers a significant improvement over Secondlook retrieval times and very long TTL retention of 6-12+ months. Data in the Warm tier is stored in “closed” indexes which have to be opened upon request.
When searching Warm (ultra or closed) tier, only one search request can be processed simultaneously. Additional Warm tier search requests are queued until the Warm tier search requests in front of them complete or the search timeout is expired for searches across the Warm (closed) tier searches can take some time.
Warm (closed) tier uses a mechanism of opening/closing indexes when searches are performed. Indexes are opened/closed in chunks of days; depending on the duration of the search, this can mean many rounds of opening/closing if the search is over an extended period of time. As of LogRhythm 7.19, this value is configurable; prior versions were hard-coded to five days. It is recommended setting this value to no more than 30 days.
User initiates a search of data from Warm Tier through Web Console or Client Console.
Colombo on the DX receives the request and detects the search request spans Warm tier. Since only one Warm tier search can run concurrently, Colombo initiates a lock.
Colombo closes all open indexes in the Warm tier which do not fall within the search duration dates.
Colombo search:
Open five indexes,
Search the five indexes,
Return the results to the user from those five indexes,
Closes the 5 indexes, and
Repeat steps a through d until the duration of the search request is satisfied or the max results are hit, whichever comes first. For example, a search of 30-day Warm data will go through six cycles if it does not hit max results.
Colombo releases the Warm tier search lock.
Opening and Closing of indexes cause the cluster to temporarily go “red.” This is normal behavior when Warm (closed) tier searches are run. While the cluster is red, normal indexing and searching of the Hot tier still functions; however, this performance may be slightly degraded. Some cluster commands are not accepted (like creating a new index), but this should have no noticeable impact to the user.