Configure Beats for JSON Parsing

By default, the Open Collector uses the JQ scripting language to process logs and send them to data processors or hosts, which can sometimes require multiple Open Collectors to retrieve all of the necessary log data. However, Beats can also be configured to use a JSON parsing engine on a System Monitor for log processing, which improves processing performance, allows for more logs to be processed through a single Open Collector, and creates a simpler administrative workflow.

Every Beat configuration now includes the option to enter a System Monitor's hostname or IP address and a port that can be used to bypass the JQ-based parsing engine and use the new JSON parsing engine instead.

Prerequisites

• This feature is available for customers who have upgraded to LogRhythm SIEM version 7.13 with System Monitor version 7.13.

  

  Existing Beats are not required to be upgraded to the new JSON parsing method.

• System Monitor running on Windows 64-bit with a minimum of 8 CPUs and 16GB RAM.
• Ensure TCP port 5044 is open between the Open Collector and System Monitor hosts.

Enable JSON Parsing on System Monitor Agents

In order for an Open Collector to send JSON parsing directly to a System Monitor, that System Monitor must first have JSON Parsing enabled.

Enable JSON Parsing for a New System Monitor Agent

To create a new System Monitor with JSON parsing enabled, from the Client Console:

1. On the main toolbar, click Deployment Manager.
2. Click the System Monitors tab.
3. On the File menu, click New.
   The System Monitor Agent Properties window appears, and the Agent Settings tab is selected.
4. Configure the System Monitor's basic or advanced properties as desired.
5. On the Syslog and Flow Settings tab, check Enable Syslog Server and then check Enable Json Parsing.

6. In the Syslog Relay Regular Expressions box, add the following regular expression to the top:

   CODE

   <(?<priority>\d{1,3})>\s*(?<month>\d{2})\/(?<day>\d{2})\/(?<year>\d{4})\s(?<hour>\d{2}):(?<minute>\d{2}):(?<seconds>\d{2})\s(?<message>.*)

   

   For Agents running on version 7.18 or later, use the following regex instead:

   CODE

   ^<(?<priority>\d{1,3})>\s*(?<year>\d{4})-(?<month>\d{2})-(?<day>\d{2})T(?<hour>\d{2}):(?<minute>\d{2}):(?<seconds>\d{2})(\.(?<ms>\d+))?Z?[-+]?[0-9:]{0,}\s.*

7. When you are finished, click OK.

Enable JSON Parsing for an Existing System Monitor Agent

To modify the properties of an existing System Monitor to enable JSON parsing, from the Client Console:

1. On the main toolbar of the Client Console, click Deployment Manager.
2. Click the System Monitors tab.
3. Select a System Monitor to modify.
   You can use the filters at the top of grid to search for a System Monitor.
4. Double-click the System Monitor you want.
   The System Monitor Agent Properties window appears.
5. Click the Syslog and Flow Settings tab.
6. Check Enable Syslog Server and then check Enable Json Parsing.

7. In the Syslog Relay Regular Expressions box, add the following regular expression to the top:

   CODE

   <(?<priority>\d{1,3})>\s*(?<month>\d{2})\/(?<day>\d{2})\/(?<year>\d{4})\s(?<hour>\d{2}):(?<minute>\d{2}):(?<seconds>\d{2})\s(?<message>.*)

8. When you are finished, click OK.

   

   You cannot change the entity to which a System Monitor belongs. For information on what to do in this case, see Connect System Monitor Agents to Different Entities.

Enable JSON Parsing During Configuration of a New Beat

Once the System Monitor Agent has JSON parsing enabled, the Beat can be configured to send logs directly to that agent.

To enable JSON parsing for a Beat:

1. Follow Beat configuration guides as normal to complete the Beat configuration and log source setup in the client console.
   Refer to the table in the Beat Initialization Guides Reference section below to find the configuration guides for each Open Collector Beat.

2. During the initialization steps, provide the Hostname or IP address of the System Monitor, and then press Enter.
   

   

   
   This should be the hostname or IP address of the System Monitor Agent configured in the previous section.
3. Enter the port number for data collection.
   

   

   
   By default, the port is 5044.

4. Continue initializing the beat as normal.

Reconfigure an Existing Beat to Use the JSON Parsing Engine on a System Monitor

To repoint an existing Beat from the JQ engine on an Open Collector to the JSON engine on a System Monitor, update the Beat configuration using the steps below. After rerouting an existing Beat to the System Monitor's JSON engine, logs will continue to be parsed and associated with the existing log source in the SIEM.

First, ensure that your System Monitor agent has been configured for JSON parsing using the Enable JSON Parsing on System Monitor Agents section.

From the Open Collector host running the Beat you wish to update:

1. Run the following command to view the current Beat configuration.

   CODE

   ./lrctl [Beat Service Name] config view

   For example:

   CODE

   ./lrctl oktabeat config view

2. Select the Beat to be reconfigured using the arrow keys, and press Enter.

3. Note the Beat configuration.

4. Run the following command to edit the Beat configuration:

   CODE

   ./lrctl [Beat Service Name] config edit

   For example:

   CODE

   ./lrctl oktabeat config edit

5. Select the Beat to be reconfigured using the arrow keys, and then press Enter.

6. Confirm and re-enter previously configured settings such as the unique identifier, domain, API key, etc. until asked for the hostname or IP address of the Open Collector.

   

   The configuration settings vary from Beat to Beat. Refer to the Supported Beats and Initialization Guides Reference to find the initialization steps for your Beat.

7. Enter the hostname or IP address of the System Monitor agent configured in the Enable JSON Parsing on System Monitor Agents section and press Enter.
   

   

8. Enter the port number for data collection and press Enter.
   

   

   
   By default, the port is 5044.

9. Restart the Beat.

   CODE

   ./lrctl [Beat Service Name] restart

   For example:

   CODE

   ./lrctl oktabeat restart

10. Once the beat has restarted, to verify the changes, view the configuration using the following command:

    CODE

    ./lrctl [Beat Service Name] config view

    For example:

    CODE

    ./lrctl oktabeat config view

Supported Beats and Initialization Guides Reference

Refer to the table below to find the list of beats that currently support JSON parsing, as well as the initialization guides for those beats.