Data Archives and Restoration

The LogRhythm Mediator Server service is responsible for archiving specified log data from active indexes to the LogRhythm Archives. The integrity of LogRhythm archives are protected during their various stages of processing through file attribute monitoring and SHA1 hashing. Archive file attributes, hashes or both are recorded by the LogRhythm system for use in verifying integrity during archive restoration and other operations. The Mediator Server uses configuration parameters that control the way the deployment archives data.

When you need access to archived logs, the SecondLook Wizard allows you to import them into a special archives index.

Active vs. Inactive Archives

The contents of active and inactive archive files are the same: original log data and metadata generated during LogRhythm’s log processing. Active and inactive archive files are written to separate directories to ease backup of sealed archive files. The differences between the two are:

Active archive files have not reached the maximum size allowed and are still in the process of having data written to them. The maximum size is configurable.
Inactive archive files have been sealed.
Sealed archive files have been hashed for data integrity verification and compressed for storage.

For best performance, keep inactive archives on a file share other than the LogRhythm appliance.

When using an external path for inactive archives, the account under which the Mediator service is running must have delete permissions on that path. If the Mediator does not have delete permissions, you may see "path is not writable" errors in the Mediator log.

Archive File Structure

LogRhythm archive files are binary files with a formatted file name. Uncompressed archives end with the file name extension .LUA and compressed archives end with the file name extension of .LCA.

A LogRhythm archive has the following naming scheme:

LogRhythm archives are placed in the folders specified by the Mediator Server configuration.

Additionally, inactive archives are placed inside folders, grouped by day, for convenience. Inactive archive directories use this naming convention: YYYYMMDD_MediatorID_Ticks where ticks = milliseconds since 12:00:00 midnight, January 1, 0001.

Archive File Protection

Archive file protection refers to the method used to ensure the integrity of the contents of an archive file. During writes to the active archives, the archiver applies the appropriate level of file protection to ensure it is the only process that has written to, or altered, the file. Inactive file protection allows the archive file to be verified prior to restoration to ensure the integrity of the log data contained therein. Active and inactive archives have configurable levels of protection. The file protection levels are governed by the Mediator Server configuration.

The following protection levels are available:

None. No protection used. Fastest, least computationally expensive option.
File Attribute Tracking. The file size and last modification date are tracked by the archiver, to ensure it is the only process that has written to the file. Good performance, with lightweight data integrity verification.
SHA1 Hashing. A SHA1 hash of the file is calculated. Most computationally expensive, but most reliable method for data integrity verification.

By default the archiver uses File Attribute Tracking on the active archive files, and SHA1 Hashing of the inactive archive files. This configuration offers the most balanced combination of computational efficiency, and file protection. The File Attributes or SHA1 Hash of each archive file is captured and stored in the Platform Manager database.

It is highly recommended that SHA1 Hashing protection not be used on the active archives, unless significant CPU and IO resources are available on the system running the Mediator service.

Archive Sealing

When an active archive is sealed, and becomes an inactive archive, the following steps are taken:

The final file protection parameters are gathered for the active archive (File Attributes or SHA1 Hash), if archive file protection is enabled.
The final file protection attributes are recorded in the LogRhythm Platform Manager database.
The archive is compressed, if compression is enabled.
The active archive is moved to the inactive directory and into that day's folder.

Maintenance Process

The Mediator Server handles archiving of log data to LogRhythm archive files, while Elasticsearch on the Data Indexer handles the deletion of expired log data and index maintenance.

In addition to archiving, there are SQL Server stored procedures that handle database maintenance on the LogRhythm Platform Manager. Some have parameters that can be configured to help prevent SQL deadlocks. For example, with the LogRhythm_Events_PartitionMaintenance_Msg procedure, increasing the Retry Count and decreasing the Wait Interval can shorten the elapsed time for the maintenance job to run. For more information, contact LogRhythm Support.

To automate the maintenance, the stored procedures are scheduled to run as SQL Server jobs which, by default, run daily at 12:00 AM.