Message Processing Engine Rule Builder

You use the Message Processing Engine (MPE) Rule Builder to view, create, and edit new MPE base rules and sub-rules. New rules are needed to collect and process logs from any new Log Source Type.

Training for Rule Building

You can submit a Device Support Request to LogRhythm to have new rules developed by LogRhythm Engineers. Alternatively, you can build MPE Rules yourself; however, it is highly recommended that you attend the rule building training offered by LogRhythm before you attempt to build custom rules. Customers who attend LogRhythm Custom Rules Training have the option of taking the additional two-day Advanced Rule Development class. The class includes hands-on, instructor-led training and class materials that instruct you on how to create rules for your organization. This topic presents a brief overview of the MPE Rule Builder, but is not intended to be a substitute for the Advanced Rule Development training.

To find out more about available training schedules, contact your LogRhythm sales representative.

MPE Policies

An MPE Policy is a collection of MPE Rules designed for a specific Log Source Type such as Cisco PIX or Windows 2003/2005 Security Event Log. Only Logs that are generated from Log Sources that have an assigned MPE Policy are processed by the MPE.

Log Sources can only be:

• Assigned an MPE Policy associated with the same Log Source Type.
• Assigned an individual MPE Policy.

The LogRhythm Knowledge Base comes with pre-packaged default MPE Policies for all supported devices. It is the MPE Policy, not the MPE Rule, that determines the data management and event management settings. The MPE Rule sets the default settings for the policy when the Rule is first added.

Custom Policies can be tailored to the type of system to which they are assigned. For example, event forwarding settings for security event logs on a file server could be different from those on a domain controller.

You can download MPE rules by going to the LogRhythm Community, clicking on the Shareables link on menu at the top of the page, and then click on the Log Sources option. The filters allow you to choose from supported and unsupported plugins, as well as ones created by LogRhythm or by other users.

Base Rules and Sub-Rules

The rules used for identifying and processing log messages include:

• Base Rule. Contains a tagged regular expression (the regex) used to identify the pattern of a log and isolate interesting pieces of metadata. Using a tagging system, these metadata strings can be directed to special fields used by LogRhythm to better interpret a log or specifically identify it. In general, base rules identify log messages by matching the fields to a specific log format or pattern.
• Sub-rule. Differentiates log messages that match the same base rule using values in the log. Sub-rule tags can include a regex that only applies to the string in a specific field to identify information such as a log / event identification number, a message string, or even a user or group name.

LogRhythm MPE rules use .NET/C# syntax for writing regular expressions.

Examples

Here is an example of a log message that might be received via syslog depicting a Microsoft SQL Server authentication.

6/18/2007 2:36 PM TYPE=Information USER=NT AUTHORITY\SYSTEM COMP=DBSERVER SORC=MSSQLSERVER CATG=Logon EVID=17055 MESG=18453 : Login succeeded for user ''NT AUTHORITY\SYSTEM''. Connection: Trusted.

The base rule matching this log is named SQL Server Authentication, and has the following regex:

COMP=(?<dname>[^\s]+)?\s+SORC=mssqlserver\sCATG=.+?EVID=17055\s+MESG=(?<tag1>

18453|18454|18455|18456).+?user\s\'([^\\]+?\\)?(?<tag2.login>[^\']+)\'.*+?\\)

?(?<tag2.login>[^\']+)\'.*

This base rule matches the pattern of the log and pull the metadata textDBSERVER into the <dname> tag, 18453 into <tag1>, and NT AUTHORITY\SYSTEM into <tag2.login> (which puts the metadata into both the <tag2> and <login> fields).

The sub-rule matching the metadata in those tags is then MS-AppLog Msg 18453: Successful Authentication, which matches when 18453 is in <tag1>, and anything (*) is in the <tag2> field. This sub-rule specifically identifies the occurrence as a successful authentication to a SQL server.

Rule Processing Logic

Rules are evaluated in the following order:

• Custom base rules are always run before system base rules
• Custom sub-rules are always run before system sub-rules
• Sub-rules where VMID should equal a specific value are always processed before sub-rules where VMID can be any value including system sub-rules

Processing logic for a log:

• Evaluate log against base rule, if matched.
• Evaluate log against all sub-rules:
• If sub-rule is matched, associate log with sub-rule.
• If no sub-rules or no sub-rule match, associate log with base rule unless base rule is a pattern.

Pattern rules:

• Use in instances where a log should only match a sub-rule.

• To specify a base rule as a pattern, prefix the rule name with “Pattern”.

  

  Pattern rules are processed differently in the Test Center. To test such rules, rename them without using the prefix "Pattern." Add the "Pattern" prefix back when you have finished testing.

Base Rule Sorting

MPE Base rules are auto-sorted and have the following options

• Customers can specify static or auto-sorting for each custom base rule.
• Customers can enforce relative ordering for each auto-sorted custom base rule.
• Customers can override the relative ordering for each auto-sorted system base rule.
• Customers can specify that an auto-sorted custom rule should sort above all system rules.

System sub-rules and custom sub-rules are statically sorted.

The rule processing order for Base Rules is:

• Custom Base rules by Sort Order
• Custom Sub-rules where the value parsed for VMID should be a specific value (i.e., VMID=1001, VMID=1002…) by sort order.
• All other Custom Sub-rules by sort order.

System Base rules by Sort Order:

• Custom Sub-rules where the value parsed for VMID should be a specific value (i.e., VMID=1001, VMID=1002…) by sort order.
• System Sub-rules where the value parsed for VMID should be a specific value (i.e., VMID=1001, VMID=1002…) by sort order.
• All other Custom Sub-rules by sort order.
• All other System Sub-rules by sort order

Sub-Rule Sorting

When using wildcards and Regexes, it is important to understand how their sorting is treated by the MPE. The sort order determines the priority in which a sub-rule is matched. For instance, if you have a sub-rule where all the mapping tags are wildcards, you want to make sure this sub-rule is the last sorted item, because sorting it higher would cause it to always match, and rules below would never be tested for matching. Sorting only affects sub-rules with wildcards and Regexes. Sorting is irrelevant for sub-rules where each mapping tag value is specified since the sub-rule only matches the exact values specified.

The MPE processes rules in the following order:

1. Custom sub-rules where VMID is equal to a specific value.
2. System sub-rules where VMID is equal to a specific value.
3. Custom sub-rules where VMID is NOT equal to a specific value.
4. System sub-rules where VMID is NOT equal to a specific value.

MPE Rule Statuses

There are three statuses used in the MPE Rule Builder, which can determine how created MPE rules are applied to log messages.

Status	Description
Development	While creating new rules, use the Development status to ensure they are not used to identify log messages while being built.
Test/Production	This allows the rule to be applied to log messages by the MPE in the LogRhythm SIEM. Previously, the Test status would not trigger alarms and would automatically disable rules after 1000 log messages were processed. However, the Test status is currently functionally identical to the Production status. MPE rules that are new and undergoing testing can be given the Test status, and once the rules have passed testing and are ready for everyday use, they can be given the Production status. The logs that match the MPE Rule can be displayed on the Dashboard. Custom MPE rules can be edited, even in Production mode; however, System MPE Rules (created by LogRhythm) cannot be edited. Review the scmedsvr.log file for messages about performance issues.

Parse Fields and Tags

The following tables provide lists of all the metadata fields LogRhythm can parse, as well as their associated parsing tags, and default regex. The fields are grouped by how they appear in the Web Console. If you do not see a field in the Web Console in the same tab as this document, you may have tagged the field as a favorite, in which case the field will appear in the Favorites tab instead of the main group tab as shown in this document. If necessary, the default regex can be overridden, as described in Override the Default Regex.

All Mapping and Parsing tags are lower case.

Map Tags

Five additional tags are available for identifying data in the log specifically for sub-rules. These tags do not parse text into metadata fields, so they do not appear in Investigations, Reports, and so on. These tags are intended only to identify portions of the log message that should be used in the development of sub-rules.

Override the Default Regex

The default regex is applied by using only the named group tag. For example, <account> will apply the regex pattern \w+ in the rule, as shown in the table above.

If the default regex for a parsing tag will not properly parse the correct data out of the log message or is not the optimal regex from a performance perspective, the default should be overridden. To override the default regex, the following syntax should be used:

(?<[tagname]>[regex])

For example, suppose your regex needs to match file names with a specific extension such as the sample log message below:

User john.doe opened AnnualReport.pdf

If the base rule was written as:

User <login> opened <object>

The value parsed for login would be john and the value for object would be AnnualReport. This is due to the fact that a period is not a word character and the default regex of “\w+” would only match up to the period. Instead, the default regular expressions should be overridden, and the base rule should be:

User (?<login>\w+\.?\w*) opened (?<object>\w+\.pdf)

Now, the base rule will parse anything for login starting with a word character that optionally contains a period followed by additional word characters.

Do not override the default regex for fields which parse an IP address, such as <sip>, <dip>, <sipv6>, and so on.