Configure the Kafka System Log Beat Log Source in SIEM

This section provides instructions for configuring the Kafka System log source in the LogRhythm SIEM using the log source virtualization template.

The Open Collector sends the output of every Beat to the Agent in a single syslog stream. The parent log source is a generic type: "Syslog - Open Collector." A log source virtualization template included with the LogRhythm Knowledge Base (KB) creates child log sources for each beat.

Prerequisites

• LogRhythm Client Console
• LogRhythm Administrator Account

• Open the following port:

  Direction	Port	Protocol	Source
  Outbound	443	HTTPS	kafkaBeat

• Download the latest KB .lkb file:

1. 1. From a computer with Internet access, log on to the LogRhythm Community.
   2. On the menu at the top of the screen, select Documentation & Downloads.
   3. Go to the KB tab, and then select Request KB File. 
   4. Enter the correct License ID, Deployment ID and Product Version. 
      

      

      
      
   5. Click Get Knowledge Base.
   6. Save the Knowledge Base .lkb file and transfer it to the computer on which the Knowledge Base is being updated.

Step 1: Import the Knowledge Base

To import the latest downloaded KB:

1. In the Client Console, from the Tools menu, select Knowledge, and then select Knowledge Base Manager.

   

   The Deployment Manager must be closed to access the Knowledge Base Manager.

   The Knowledge Base Manager appears.

2. From the Knowledge Base Manager, click File, then click Import Knowledge Base File.
3. Select the latest downloaded Knowledge Base .lkb file, and click OK.
   The Knowledge Base Import Wizard appears and starts unpacking and validating the Knowledge Base file. The file is checked for compatibility with your current deployment and is prepared for import. This may take several minutes. Upon completion, the Unpack Progress: Knowledge Base unpacked message appears.
4. To import the Knowledge Base, click Next.
   Upon completion, the Import Progress Import Completed message appears.
5. Click OK.
   The Knowledge Base Updated message appears.
6. Click OK.

7. On the Knowledge Base Import Wizard, click Close.

Step 2: Verify the Log Source Virtualization Template

Currently, we do not support Kafka Beat log parsing and do not require a separate log source for log parsing. Instead, Kafka Beat logs are parsed using parent log source Syslog - Open Collector. Kafka heartbeat logs are parsed using log source Syslog - Open Collector - KafkaBeat Heartbeat. Users only need to verify that Syslog - Open Collector and Syslog - Open Collector - KafkaBeat Heartbeat log sources are available.

1. After importing the latest KB, on the main toolbar, click Deployment Manager.

2. On the Tools menu, select Administration, and then click Log Source Virtualization Template Manager.

3. Verify the Open Collector template is already there in the Log Source Virtualization Template Manager, and verify the number of virtual log sources.

   

4. In the lower-left corner click Virtual Log Source Manager, and verify the virtual log sources you need are on the list.
   
   • Syslog - Open Collector
   • Syslog - Open Collector - KafkaBeat Heartbeat

5. Click OK.
   The Virtual Log Sources are available and verified.

6. Click Close.

Step 3: Configure the Syslog Relay

This step explains how to configure the Syslog Relay. The Open Collector needs Syslog Relay for the following reasons:

• By default, the agent timestamps syslog messages as they come in. The timestamp in the SIEM should reflect when the log was generated, not when the agent received this log.
• An additional Syslog Relay Regular Expression is required to correctly extract the timestamp.

Beats configured using the JSON parsing method should use the regex relay outlined in the Configure Beats for JSON Parsing topic and skip this step.

To configure the Syslog Relay:

1. Go to the System Monitors tab.
2. Double-click the agent to which you will send the Open Collector syslog.
3. Go to the Syslog and Flow Settings tab.
4. Select the Enable Syslog Server check box, if it is not already selected.
5. Type the Open Collector IP Address in the Syslog Relay Hosts field on the left.

6. Type the following regular expression as the first line in Syslog Relay Regular Expressions:

   CODE

   ^<(?<priority>\d{1,3})>\s*(?<message>(?<year>\d{4})-(?<month>\d{2})-(?<day>\d{2})T(?<hour>\d{2}):(?<minute>\d{2}):(?<seconds>\d{2})(\.(?<ms>\d+))?Z?[-+]?[0-9:]{0,}\s.*)

7. Click OK.

Here is an example of a configured Syslog Relay, where the Open Collector IP address is 10.3.0.1.

Step 4: Accept the Pending Log Source

After Open Collector logs are sent to the Windows System Monitor Agent, you need to accept the pending log source.

1. Click the Log Sources tab.
2. In the New Log Sources grid, select the Action check boxes for the following:

   • Log Source Type. Syslog - Open Collector

   • Log Processing Policy. LogRhythm Default

3. Right-click the selection, click Actions, and then click Accept.

4. Select one of the following:

   • Customize and change the following as needed:
     

     • Collection System Monitor Entity

     • Log Message Processing Settings

     • Log Data Management and Processing Settings

     • Silent Log Message Source Settings

   • Default to select customized defaults that were previously selected. 

   • Select a default batch amount between 100 and 5000.

5. Click OK.
6. Click Refresh to see the newly accepted Log Source in the grid.

Step 5: Apply the Log Source Virtualization Template for KafkaBeat Heartbeat Messages

Use the log source virtualization template imported in step 1 to create a log source specifically for Kafka Heartbeat logs.

1. Double-click the newly accepted Open Collector Log Source.
   The Log Message Source Properties window appears.
2. Click the Log Source Virtualization tab.
3. Select the Enable Virtualization check box.
4. Click Create Virtual Log Sources.
   The Create Virtual Log Sources dialog box appears.
5. In the Log Source Virtualization Template menu, select Syslog - Open Collector - KafkaBeat Heartbeat
6. Click Save.
   The confirmation prompt appears.
7. Click OK.
   New Log Sources appear in the grid as children of your parent log source.