Configure Support for Multi-Line Logs

Multi-line log support permits the System Monitor Agent to identify and collect log entries file that span multiple lines in a text file. The System Monitor Agent preserves all formatting contained in the multi-line log, such as tabs. Multi-line log collection is supported in the Windows, Linux, Solaris, AIX, and HP-UX agents.

The Log Source supports three configuration parameters:

• Log Message Start Regex
• Log Message Delimiter Regex
• Log Message End Regex

These parameters serve two purposes:

• To identify the log entry as a multi-line log format by configuring at least one of the parameters.
• To identify the Regex strings required to delineate the multi-line log entries.

Other considerations

• The Data Processor can store logs up to 65535 characters long. When that limit is exceeded, the additional text is truncated from that single log entry.
• The delineations between log entries must be consistent throughout the log to ensure each multi-line log entry is properly collected.
• The Regex language supported for the Log Message Start Regex, Log Message Delimiter Regex, and Log Message End Regex configuration parameters varies by platform.
  • Windows. Supports perl type Regex strings with associated macros.
  • Linux and versions of UNIX. Supports posix type Regex strings with associated macros.

Configuration Parameters

Usually it is only necessary to specify one of the following parameters, depending on which one offers the simplest Regex.

• Log Message Start Regex. The Log Message Start Regex indicates the beginning of a multi-line log entry. If a line read from a log file matches this Regex string, it indicates to the Agent the beginning of a new log entry. The line matched by the Log Message Start Regex is included in the log entry.
• Log Message Delimiter Regex. The Log Message Delimiter Regex indicates the current line delimits log entries. If a line read from a log file matches this Regex string, it indicates to the Agent that the previous log entry is complete and a new log entry follows on the next line. The line matched by the Log Message Delimiter Regex is discarded and not included in any log entry.
• Log Message End Regex. The Log Message End Regex indicates the end of a multi-line log entry. If a line read from a log file matches this regex string, it indicates to the Agent the end of the current log entry. The line matched by the Log Message End Regex is included in the log entry.

Examples

The following samples are from the Snort IDS (Intrusion Detection System) Full Alert log - a multi-line log source.

[**] [1:254:4] DNS SPOOF query response with TTL of 1 min. and no authority [**]
[Classification: Potentially Bad Traffic] [Priority: 2] 
01/11-09:59:15.572292 10.1.1.10:53 -> 10.1.1.50:32769
UDP TTL:128 TOS:0x0 ID:9027 IpLen:20 DgmLen:83
Len: 55

[**] [1:254:4] DNS SPOOF query response with TTL of 1 min. and no authority [**]
[Classification: Potentially Bad Traffic] [Priority: 2] 
01/11-10:03:18.273166 10.1.1.10:53 -> 10.1.1.50:32769
UDP TTL:128 TOS:0x0 ID:13347 IpLen:20 DgmLen:83
Len: 55

[**] [1:254:4] DNS SPOOF query response with TTL of 1 min. and no authority [**]
[Classification: Potentially Bad Traffic] [Priority: 2] 
01/11-09:59:15.572292 10.1.1.10:53 -> 10.1.1.50:32769
UDP TTL:128 TOS:0x0 ID:9027 IpLen:20 DgmLen:83
Len: 55[**] 

[1:254:4] DNS SPOOF query response with TTL of 1 min. and no authority [**]
[Classification: Potentially Bad Traffic] [Priority: 2] 
01/11-10:03:18.273166 10.1.1.10:53 -> 10.1.1.50:32769
UDP TTL:128 TOS:0x0 ID:13347 IpLen:20 DgmLen:83
Len: 55

Each log entry starts with the following characters, [**], and is separated by a blank line. You could use either the Log Message Start Regex parameter or the Log Message Delimiter Regex parameter, both shown below.

Log Message Start Regex: \[\*\*\] OR Log Message Delimiter Regex: ^$

The escape (backslash) character (\) precedes the [ and * characters because they are reserved Regex characters. A simple and clear Regex to use would be the Log Message Delimiter Regex: ^$.