SecondLook

The LogRhythm SIEM solution, like other SIEMs, attempts to strike a balance between the relatively high cost and performance of online storage of useful log data for quick retrieval, and the more cost-effective method of storing flat archived log data. The SIEM processes log data and writes data to both the Data Indexer and the archiver concurrently. However, after log data is made available online, it expires out of the indexed persistence layer in somewhere between 30 and 90 days, depending on your configuration, and then the only option for recovery and analysis of older log data is to perform a SecondLook Restore from the LogRhythm SIEM. This process reads in relevant archive files and processes the contents based upon the filters set in the SecondLook search initiated by the end user. The pertinent log data is then stored in the archive indexer for search and analysis.

SecondLook is always available for LogRhythm Administrators. For other users, SecondLook is only available if Allow use of SecondLook is selected in the user's profile.

The SecondLook Process

The SecondLook Restore is launched from a LogRhythm Client Console, which performs all the disk retrieval of archive files, log processing of raw log data, and indexer inserts of retrieved data. The actual SecondLook Restore process consists of the following underlying steps:

1. After the parameters of the SecondLook Restore are defined and the restore job is executed, the Client Console starts a new instance of the Message Processing Engine (MPE) that uses the selected MPE policy selected for each target log source.
2. The Client Console searches the provided archive paths for relevant archive files, based on the chosen log source and date/time parameters of the restore.
3. When the SecondLook Restore job finds a matching archive file, the Client Console performs the following:
   1. Runs a SHA1 hash verification of the archive file (if applicable) to cross reference against the hash value of that archive file stored in the EMDB. This is to ensure that the archive file has not been tampered with since it was sealed by the archiver.
   2. Decompresses the raw log contents of the file.
   3. Submits the raw log contents to the MPE instance.
   4. Processes the raw log with the selected MPE policy.
   5. Inserts any raw log matches for provided filters (user, host, etc) to the archive indexer.
4. The previous step repeats until the client console has identified ALL archive files that meet the date/time and log source criteria.
5. The SecondLook Restore job will terminate and display relevant statistics surrounding the restore job, including number of archive files accessed, raw logs processed, and logs restored that met criteria, plus any failures encountered throughout the job.

SecondLook Best Practices

The following list are best practices for utilizing the SecondLook Wizard to minimize time it takes to restore the logs needed.

• A SecondLook Restore should not be performed on an existing Data Processor server. When it is, the two MPE instances compete for CPU resources.
• Initialize the SecondLook Restore with the most precise date/time range and log source list as possible that will meet your search criteria.
  • The process must search though all archive paths and files using date range and log source ID as a primary filter.
  • Include/Exclude Filters such as user or host information only filter what logs are restored to the archive indexer, but they must still be processed by the MPE, which also happens to be the most expensive step of the SecondLook Restore process.
• Copy the Inactive Archive files (.LCA files) for the time period you want to the server that will be used for the SecondLook restore.
  • Local storage yields faster retrieval of archive files than searching archive files across a network drive, especially when the search scope includes several hundreds of thousands of potential matches.
• Use the name of the file/folder for the timestamp of the relevant logs and NOT the date/time of the file itself. The file/folder name contains the normalized datetime of the logs within that file.
• Log in to the Client Console as LogRhythmAdmin or another user with the role of GlobalAdmin when performing the SecondLook restore, and ensure that the Windows user account under which the LogRhythm Client Console was launched is a local administrator.
  • Doing so ensures that the Client Console has adequate access to all local files/paths that contain archive files.
• If the restore period is longer than a week's worth of data, perform multiple SecondLook restores with concurrent date ranges. 
  
  • This range recommendation is variable in nature, and is directly related to the amount of log data that must be searched.
  • Doing so keeps the process from running over several hours or days even, and raises the chance of completion so that the process does not have to be restarted.
  • Currently, there is no way to "restart" a SecondLook restore, so if it does not run to completion, then there is no way to guarantee that log data was found by the previous SecondLook Restore attempt.
• Run extremely high volume deployments and/or large SecondLook timeframes, or multiple concurrent SecondLook Restores, on separate servers.
  • Doing so provides increased throughput of the SecondLook restore process. The workloads can be divided using log sources, date/time range, or both.
  • This approach works best when both SecondLook instances are pointed to the same archive file location. While this point does contradict previous the suggestion of copying archive files for local access, this approach yields more throughput at the MPE processing stage of a SecondLook Restores, which is more of a performance bottleneck than the file search process.
• Tune the MPE settings of the Archive Data Processsor to use the following enhancements: LogProcessingThreads = # of physical cores minus 2 (a 32 core system would yield a thread count of 30)
• The MPE policy used for a SecondLook Restore should be as "skinny" as possible.
  • The SecondLook Restore process is mostly constrained by the MPE processing of the LogRhythm client console.
  • MPE policy should contain as few rules as needed to identify the logs that are the target of the SecondLook Restore. This keeps the MPE performance high by not utilizing unnecessary MPE rules that do not match on relevant logs.
  • Individual MPE rules should parse as few data elements as possible while still yielding useful and indexed results. For instance, if the SecondLook Restore is meant to find user accounts, then the MPE rule should only parse perhaps the Event ID and the user field from the raw log message.
• Disk I/O, both network shares and local disk arrays, did not come into play as evidenced by performance counters, and should be considered a secondary performance consideration behind MPE optimization.
• Perform parallel restores using separate servers with dedicated resources for MPE processing, with restore jobs logically segmented by timeframe yields scalable results.