Advanced Intelligence Engine

LogRhythm AI Engine detects conditions in your deployment that occur over multiple data sources and time ranges. AI Engine can connect the dots to paint a picture that is far more enlightening than its individual parts and reveal potential problems while you still have time to take effective action. It provides realtime visibility to risks, threats, and critical operations issues. There are more than 100 preconfigured rule sets that can be used in the wizard-based drag and drop GUI.

For information on the Risk-Based Priority calculations used with AI Engine Rules, see Global Risk Based Priority.

The AI Engine components include:

• AI Engine (service on the AI Engine server)
• AI Engine Communication Manager (service on the AI Engine server)
• AI Engine Data Provider (within the LogRhythm Mediator)

AI Engine Log Files

There are three log files that can be reviewed for success, error, and general log messages related to the AI Engine. The number of logs you see depends on the LogLevel that is set in the AI Engine configuration file.

• The LRAIEComMgr.log and LRAIEEngine.log files are on the AI Engine Server in the location selected during installation. The default location is C:\Program Files\LogRhythm\LogRhythm AI Engine\logs
• The LRAIEDP.log file is on the Data Processor Server in the location selected during installation of the mediator. The default location is C:\Program Files\LogRhythm\LogRhythm Mediator Server\logs

AI Engine System Rules

There are several system rules that come packaged with the AI Engine software. The rules must be downloaded and imported using the Knowledge Base Import Wizard. For more information, see Import a Knowledge Base.

AI Engine Diagnostic Alarms

The following table lists the AI Engine Diagnostic Alarm Rules and their properties.

AI Engine Communication Manager Performance Counters

The AI Engine Installer installs performance counters for the AI Engine Communication Manager. The Performance Category is called LogRhythm AI Engine Communication Manager, and it consists of the following counters:

The Mediator Installer installs performance counters for the AI Engine Data Provider. LogRhythm AI Engine Data Provider consists of the following counters:

AI Engine Components

AI Engine

The AI Engine runs the Windows Service LRAIEngine. It receives the logs sent from the AI Engine Communication Manager and applies the AI Engine Rules to process the logs. It generates Events when those rules are satisfied. The AI Engine also provides diagnostic performance data and can save and reload state when it is shut down and restarted.

The AI Engine design uses time-binned aggregated data when it processes logs against the AI Engine rules. Therefore, processing is not affected by the order the data arrives.

AI Engine Communication Manager (ComMgr)

AI Engine ComMgr consists of these two parts:

• AI Engine Data Provider (AIEDP) located in the Data Processor Mediator.
  • AIEDP runs on the LogRhythm Mediator and sends logs to the AIEDR in all specified AI Engine servers. It starts and stops along with the Mediator service.
  • AIEDP configuration data is on the system where the Mediator resides: C:\Program Files\LogRhythm\LogRhythm Mediator Server\config\LRAIEDP.ini

  • AIEDP writes logs to the following: C:\Program Files\LogRhythm\LogRhythm Mediator Server\logs\LRAIEDP.log

• AI Engine Data Receiver (AIEDR) located in the Communication Manager service on the AI Engine Server. 
  
  • AIEDR can be started and stopped from either the command line or from the Service Control Manager. It starts and stops along with the AI Engine ComMgr service (LRAIEComMgr).
  • AIEDR configuration data is on the system where the AI Engine resides: C:\Program Files\LogRhythm\LogRhythm AI Engine\config\LRAIEEngine.ini

Event Log

The ComMgr writes logs to the following: C:\Program Files\LogRhythm\LogRhythm AI Engine\logs\LRAIEComMgr.log

AI Engine User Interface

The AI Engine User Interface has an AI Engine Rule Manager that lists the current AI Engine Rules, and an AI Engine Rule Wizard where you create and modify rules.

These screens and other information are described in detail in the AI Engine Rule Manager and the AI Engine Rule Wizard.

How AI Engine Rules Use Time Limits

AI Engine rule blocks include user-selected time limits to define the time span within which the rule block condition must be met. In multiple-block rules, a time limit may be contingent on a preceding rule block. For example, you can create an event if rule block B (a specific application starts) occurs or does not occur within X minutes after rule block A (the nightly backup finishes) occurs.

Depending on rule properties, you can set a time limit from one of the following locations in the AI Engine Rule Wizard:

• Threshold tab
• Unique Values tab
• Rule Block Relationship window
• Rule Block Time pane

AI Engine Rule Structure

AI Engine rules are able to span multiple data sources and time ranges by using up to three separate rule blocks. The AI Engine Rule Wizard provides a means for you to create custom rules. Each rule block has its own specific properties, which are applied in the following order:

• Log Source
• Day and Time
• Primary Criteria
• Include Filters
• Exclude Filters

The Rule Block Relationship window allows you to identify the common fields shared by two rule blocks and identify any time constraints that one block imposes on the other. For example, you can instruct the second rule block to become effective one hour after the first rule block is satisfied.

Using multiple rule blocks and relationships, the AI Engine can detect situations such as:

• A server was attacked and later started transferring large amounts of data out of the network.
• A backup started but never finished.
• A critical service or system did not restart.
• Worm propagation.

For more information on AIE Rule Creation and Best Practices, search Best Practices Guide: Understand and Create AIE Rules on the LogRhythm Community.

AI Engine Rule Wizard Tabs

The AI Engine Rule Wizard consists of the following tabs:

• Rule Blocks. The Rule Blocks tab is divided into four panes:

  • Rule Block Types. Used to add new Rule Blocks to an AI Engine Rule. The types are separated into four categories:

  • Rule Block Designer. Used to edit the properties of a Rule Block.

  • Rule Block Times. Displays the time spans in which Rule Block processes log messages.

  • Summary. Displays details for the item selected in the Rule Block Designer.

• Settings. The Settings tab allows you to configure common event properties, alarm properties, and general properties for the AI Engine Rule. When the Sync with rule name box is selected, the Common Event name is synchronized with the rule name. For information on the settings you can configure on the Settings tab, see Filters—Settings.
• Notify. The Notify tab allows you to designate the Roles, People, and Groups to be notified when a rule has been satisfied. If you are using data segregation to limit alarm notifications to only people who have access to a certain entity, each recipient on this tab must have an associated user profile that grants them access to that entity. For more information, see User Profile Manager and Configure Notifications in the Filter Editor.
• Actions. For information on the settings you can configure on the Settings tab, see Filters—Action.
• Information. For information on the settings you can configure on the Settings tab, see Filters—Information.

When you access the AI Engine Rule Wizard to Create an AI Engine Rule (Complete Guide) or modify one, the Rule Blocks tab appears by default.

Types of Rule Blocks

An AI Engine rule can include up to three sub-components called rule blocks. Each rule block has its own data source, filter criteria, time frame, and conditions.

All rule blocks continuously receive logs that match their designated data source and filter criteria. The AI Engine periodically checks each block to see if its condition has been detected. When the condition is met, the AI Engine checks related blocks in the rule to see if their conditions are also met. If that happens, an event is generated. An alarm may also be generated depending on the rule configuration.

There are four basic rule block types:

• Log
• Threshold
• Unique Values
• Behavioral

Except Behavioral, each type contains three variations:

• Observed
• Not Observed Compound
• Not Observed Scheduled

The variations for Behavioral are: Whitelist, Statistical, and Trend.

AI Engine Rule Manager Grid

The Rule Manager grid includes the following columns. When changing the View from All Rules to a Server, there are some changes to the columns shown.

Column Head	All Rules	Per Server	Description
Action	Y	Y	Select one or more check boxes.
AI Engine Rule Name	Y	Y	Name of the AI Engine Rule. Max characters = 100 Because rules are identified by ID, the name is not required to be unique. However, use a naming convention that promotes clarity for your deployment. The AI Engine rule name is automatically assigned to the associated Common Event and Alarm Rule in the format AIE: followed by the first 45 characters of the rule name.
Rule Status	Y	Y	Disabled The rule is not processed. Intended for temporary usage. When you disable a rule, the in-memory log data supporting that rule is discarded. If you re-enable the rule, the in-memory log data supporting that rule begins to be accumulated again. Enabled The rule is processed as normal. Expired The rule was configured with an expiration date/time which has passed. The rule is not processed. Learning A rule with one or more applicable Behavioral Rule Blocks is accumulating state information. All data is being collected for the rule, but it is not yet being evaluated. When the Activation Date is reached, it goes into the Enabled state. Paused A rule has been paused to stop generating New events, but continues to maintain all state data for the rule. Retired The rule is not processed and by default does not appear in the grid. Intended for long-range or permanent usage. Suspended Rules with errors or which consume too many resources such as memory or CPU time are automatically suspended and are not processed. The Suspend Reason column displays more information. After diagnosing the reason for the suspension, you must re-enable the rule manually to take it out of Suspended status. When a rule is suspended, the in-memory log data supporting that rule is discarded. If you re-enable the rule, the in-memory log data supporting that rule begins to be accumulated again. Unassigned The rule is not assigned to any engine. Either the rule is not in a rule set mapped to a workload or the workload is not assigned to an engine.
Restart		Y	Displays the Restart status. blank: No Restart Required Restart Needed : Rule has been edited and requires a restart Restart Pending: Restart button pressed but restart has not yet completed
Activation	Y	Y	For Behavioral Rules with an Activation Date, indicates when the Rule goes from Learning status to Enabled status.
Expiration	Y	Y	Date the rule expires if one is set. N/A indicates that no expiration date is associated with this rule. This cell is selected for expired rules.
Data Segregation Mode	Y	Y	Disabled No Data segregation is in effect. Entity All rule processing and events are segregated by the Entity of the logs. Root Entity All rule processing and events are segregated by the Root Entity of the logs.
Alarm Status	Y	Y	Identifies the alarm status of a rule as Enabled or Disabled. An enabled rule that has its alarm status disabled generates events, but not alarms.
Auto Drilldown			If the Alarm Status column shows Enabled, this column shows Enabled or Disabled. If the Alarm Status column shows Disabled, this column shows N/A.
EDF	Y	Y	[E]nvironmental [D]ependence [F]actor is used in risk calculation and is set on Settings tab of the AI Engine Rule Wizard Tabs. It estimates how much additional configuration is required for the rule to function as expected within different network environments. Possible values: None = No additional configuration required. Low = Minimum additional configuration required. Medium = Additional configuration required. High = Significant additional configuration required.
FPP	Y	Y	False Positive Probability is used in risk calculation and is set on Settings tab of the AI Engine Rule Wizard Tabs. It estimates how likely the rule is to generate a false positive response.
Suppress For	Y	Y	The amount of time repeated events are suppressed. The event associated with this rule is not created more frequently than the value stated here once every 15 minutes for a unique occurrence. For example, if the rule detects server failures, it does not create an event more than once every 15 minutes for a particular server. It can create events more than once every 15 minutes for different servers. When the rule's alarm is enabled, it is always generated for every event.
Runtime Priority	Y	Y	Values = Low, Normal, or High. If the AI Engine begins to run out of memory, it tries to reclaim memory by first reducing excess log grace period then, second, suspending rules starting with the lowest runtime priority. Best Practice: Start new, untested rules with a Runtime Priority = Low until processing is verified. Set important, well-tested rules to Runtime Priority = High.
CPU Cost		Y	Displays the percentage of CPU this rule consumes relative to all rules in the engine.
Unshared Mem Cost		Y	Displays the percent of unshared memory this rule consumes relative to all memory consumed by all rules.
Shared Mem Cost		Y	Displays the percent of shared memory this rule consumes relative to all memory consumed by all rules.
Unshared Mem KB		Y	Displays the number of KB of unshared memory consumed.
Shared Mem KB		Y	Displays the number of KB of unshared memory consumed.
Total Mem KB		Y	Displays the total memory used in KB.
Current Event Forward Rate	Y	Y	Displays the Average event forwarding rate (events/hour) over the last 3 minutes.
Average Event Forward Rate	Y	Y	Displays the Average event forwarding rate (events/hour) over the last 24 hours.
Current Event Feedback Rate	Y	Y	Displays the Average event feedback rate (events/hour) over the last 3 minutes.
Average Event Feedback Rate	Y	Y	Displays the Average event feedback rate (events/hour) over the last 24 hours.
Rule Group	Y	Y	Name of the group to which this rule has been assigned, if any.
Description	Y	Y	The first part of the rule description. The entire description can be seen in the rule properties
Rule Set	Y	Y	Displays the name of the assigned Rule Set.
Permissions	Y	Y	System: Global Admin - A System AI Engine Rule can only be created by LogRhythm. It imports in a disabled state and must be enabled by a user that logs in with Global Admin permissions. Custom: Global Admin - a Custom AI Engine Rule can be created by a user that logs in with Global Admin permissions.
Date Updated	Y	Y	Date of last update.
Rule ID	Y	Y	A unique number generated by AI Engine and assigned to an AI Engine Rule. The first System AI Engine Rule increments from 1. The first Custom AI Engine Rule increments from 1000000001.

AI Engine Servers

LogRhythm deployments support multiple AI Engine servers providing a horizontally scalable deployment architecture for very large deployments. Each AIE node can receive all or a subset of processed log data. Each AIE node can be assigned a workload that determines which AIE rules it should run.

AI Engine Rule Sets

AIE rule sets allow for the gathering of rules into a group (set) as well as for providing a filter to specify which logs will be sent to the rule set's workload.

A workload specifies all the AIE rules to be in use by a given AIE server, and is comprised of one or more rule sets. If more than one rule set is included in a workload, any log that matches the criteria (filters) specified by at least one of the rule sets is forwarded to that workload's server and is evaluated against all rules in all of the workload's rule sets. As an example, if Rule Set A filters in only logs from Entity A, and Rule Set B filters in only logs from Entity B, and Workload C on Server C includes Rule Sets A and B, all logs coming from Entity A or B go to Server C and are evaluated against both Rule Sets A and B.

The data segregation option in an AIE rule may be used to ensure that a rule is triggered only by logs that all have the same entity or root entity. Data segregation by entity also ensures that alarm notification emails sent by the Notification Service are only sent to recipients who have access to that entity. For the Notification Service to work, the AIE Drill Down Cache must be enabled in the LogRhythm Configuration Manager and TLS 1.2 must be enabled.