System Monitor

The System Monitor is a software component that provides local and remote log data collection across various English-based operating systems including Windows and *NIX. For a complete list of supported operating systems, see the LogRhythm System Monitor Compatibility and Functionality guide.

The agent serves as a central log data collector, collecting logs from many devices, servers, databases, and applications, performing host activity monitoring and forwarding logs, via authenticated TLC connections, to the Data Processor. 

This section describes working with System Monitors in the Client Console. The System Monitor component releases on a separate schedule from the SIEM. For information on System Monitor releases and installation, see the separate System Monitor documentation.

File Integrity Monitoring

Standard and Realtime FIM are included with the System Monitor Lite license for desktop operating systems only. Server operating systems require System Monitor Pro or Collector. For a list of supported agents, see the Realtime File Integrity Monitor (FIM) Support by Operating System table.

System Monitor Functionality

To view a table that lists Agent Functionality of the 32-bit and 64-bit System Monitor Agents, see the System Monitor Functionality by Operating System.

System Monitor Agent Licensing

You must license each System Monitor to connect to a Data Processor and forward data.

LogRhythm provides the following three types of Agent licenses:

• System Monitor Lite
• System Monitor Pro
• System Monitor Collector

When a LogRhythm Agent is registered in the Deployment Manager, it is automatically assigned a System Monitor Pro license, if one is available; otherwise it is assigned a System Monitor Lite license.

To view a table that lists Agent functionality by license, see System Monitor Functionality by License: Lite, Pro, and Collector.

Supported Character Sets and Languages

With the exception of flat file collection and Windows Event Logs (see Unicode (Double-Byte) Character Support below), LogRhythm only supports ASCII and extended ASCII characters within collected log data.

LogRhythm extended ASCII support is only available for code page 1252 encoding for the following languages:

• Afrikaans
• Basque
• Catalan
• Danish
• Dutch
• English
• Faroese
• Finnish
• French
• Galician
• German
• Icelandic
• Indonesian
• Italian
• Malay
• Norwegian
• Portuguese
• Spanish
• Swahili
• Swedish

While LogRhythm accepts non-English log sources, the program is only compatible with English operating systems.

Unicode (Double-Byte) Character Support

LogRhythm supports Unicode characters when collected from Windows Event Logs and flat file Log Sources on Windows and *NIX. Search for Unicode characters is supported only in raw log and metadata searches against the Data Indexer. Currently, Unicode characters are not supported in:

• Events (PM search)
• Alarms
• Lists
• Web Console search

If you have log message sources that are not flat file sources, and these sources contain Unicode or double-byte characters, you can do the following (listed in order of preference to create the least impact when working with such sources):

• Reconfigure the logging device to create ASCII log messages only.
• Rewrite processing rules so that they only identify text up to the first non-ASCII character.
• Disable the specific rules that process non-ASCII characters.
• Disable all processing for the log source.

Tuning

Running the System Monitor initially with a LogLevel of Verbose is useful to allow you to troubleshoot and tune the System Monitor configuration parameters. The two most important parameters to tune are:

• CycleTime. The minimum time the Agent spends in a processing cycle. If the processing cycle takes longer than the CycleTime, the Agent immediately begins the next cycle. If the processing cycle is shorter than the CycleTime, the agent sleeps for the remainder of the CycleTime. This property is set in the Agent Advanced Properties window. For more information, see Modify System Monitor Advanced Properties. Default: 10 seconds.
• MaxMessageCount. Should be set to a value that keeps up reasonably with the log sources being monitored. Forwarding log data in small batches with smaller a CycleTime is generally better than sending large batches over a long CycleTime. This prevents extended periods of concentrated network traffic and is easier for the Data Processor to process. This property is set in the Message Source Properties window. For more information, see Log Sources. Default: 100.

To extract the log data in a timely manner, these parameters must be set to keep up with the source in question. It is useful to observe if the System Monitor reads its full MaxMessageCount in each cycle. This can be found in the System Monitor Agent log - scsm.log (Agent LogLevel must be set to Verbose). When the System Monitor is started on a new source, it spends some time catching up to the most recent log. After it reaches the end, if it is still reading MaxMessageCount log entries each cycle, then this parameter should probably be increased to keep up with the log.

Monitoring Intervals

If a given source is not critical for real-time monitoring, then consider scheduling it to run at a time when the log is not very busy using the MonitorStart and MonitorStop parameters in the Log Source Advanced Properties window.

Log Rotation and Compression

The Agent can follow log rotations while collecting from files, but cannot finish reading a log file that is compressed. The results of reading a compressed file are unpredictable. Disable compression of the log sources that the Agent is monitoring. Most Linux systems use the logrotate utility and its corresponding config file logrotate.conf to control this compression. See the logrotate documentation for details.

Position Files

The System Monitor produces a state tracking file (*.pos files) for each source in its configuration that it is monitoring. These files maintain position information across program shutdowns and subsequent startup. It is important not to remove or delete these files unless you really want the System Monitor to begin at the start of the log file again. The position tracking files are located in the state directory of the Agent.

Suspense File

The System Monitor spools in memory log data to a file if necessary, such as when a local shutdown is requested after having lost connection to a Data Processor. The log data in suspense.log is read by the System Monitor on startup for immediate forwarding to the Data Processor. The suspense.log file is hidden and deleted after being loaded at startup.

The Windows Agent also spools syslog data to a local file if its memory cap is exceeded. The syslog data written to file is then read and sent to a Data Processor when the Agent memory usage falls below its memory cap. The syslog suspense file is located in the state directory of the Agent. TCP suspense folder and UDP suspense folder are combined together as one single suspense folder (syslogsuspense) due to the underlying reason that TCP/UDP is a protocol/communication type and not a log source type.

Firewall Considerations

The Agent initiates all connections to the Data Processor. In order for the Agent to communicate across a firewall, a two-way TCP pinhole, or exception rule, must be opened from the Agent's host IP and Client TCP Port to the Data Processor's host IP and ServerSSLPort, and back. Client TCP Port is specified on the Data Processor Settings tab in the System Monitor Agent properties, and ServerSSLPort is specified in the Advanced Data Processor Properties window. For more information, see Modify Data Processor Advanced Properties. Agents prior to v4.0.0 connect to the Data Processor without using SSL, and instead connect to the port controlled by the ServerPort setting specified in the advanced properties of the Data Processor record.

Syslog Server Operation

The LogRhythm Windows Agent can be configured to listen on the standard syslog UDP/TCP port of 514 for incoming syslog data. When a syslog message is received, the following process occurs:

1. The time the syslog message was received by the syslog server is captured. The time is relative to the Agent receiving the syslog and is reflected by the Normal Message Date of the log. This Normal Message Date may also be corrected by the Data Processor to account for time offsets between the Agent and the Data Processor.
2. The IP endpoint and the IP address contained in the syslog packet are captured by the Agent.

   • If the IP address captured from the syslog packet is present in the Agent's Syslog Relay Hosts parameter, the Agent runs the syslog message through the list of regex strings contained in the Agent's Syslog Relay Regular Expressions parameter to parse out an IP address or host name identifier.

     

     The regex match is done against the pre-processed log with the syslog header, not against the raw log after it reaches the Data Processor.

   • If the IP address captured from the syslog packet is not present in the Agent's Syslog Relay Hosts, then the Agent uses the IP address contained in the syslog packet as an IP address identifier.
3. If present, the syslog priority is parsed from the syslog data and converted into a syslog facility and severity that is appended to the syslog message. If no syslog priority is present, the facility is set to local0 and the severity is set to information.
4. The Agent then performs a lookup into its syslog virtual message sources to see if the identifier has a virtual message source assigned and assigns the proper message source parameters (source id, TTL, and archive mode). If the Agent does not find a virtual message source, it submits a virtual source lookup request to the Mediator Server.
5. The Agent then either adds the syslog data to the message queue for delivery to a Data Processor or may spool it to the syslogsuspense directory if it cannot be delivered to a Data Processor (connection issue, unauthenticated syslog source, etc.). If the data is spooled to disk, then the agent periodically reloads this data after it is connected to a Data Processor.

Example Syslog Format

Following is an example of the syslog data received by the LogRhythm Windows Agent and the final syslog format format stored in the Data Indexer:

• Syslog data received:

  The following syslog message was received from host 192.168.1.10 on Dec 12 11:00:01 AM 2006.

  <13>Dec 12 11:00:00 This is a syslog message

  Priority = 13

  Date = Dec 12 11:00:00

  Message = This is a syslog message

• Syslog message stored in Data Processor:

  The syslog message would appear as follows in the Data Indexer:

  12 12 2006 11:00:01 192.168.1.10 <1:5> Dec 12 11:00:00 This is a syslog message

  Date = 12 12 2006 11:00:01

  Host = 192.168.1.10

  Facility = 1 (uselevel)

  Severity = 5 (notice)

  Message = Dec 12 11:00:00 This is a syslog message.