Initialize the Kafka Beat

This section provides instructions for initializing the Kafka Beat.

Prerequisites

• Open Collector is installed. If you have not already installed it, follow the instructions in the Open Collector Installation and User Guide, and then return to this topic.
• The Kafka server is running. If you have not yet setup the Kafka server, setup the Kafka server first, and then return here.
• Obtain a Kafka broker IP (IP on which the Kafka server is running).
• Obtain a Kafka broker port (Port on which the Kafka server is running).
• The Kafka topic (Topic you want to collect data from).

• CA signed certificate, if your Kafka server is running in SSL mode with a CA signed certificate and requires the beat to perform host verification.

• The following port is open:

  Direction	Port	Protocol	Source
  Outbound	443	HTTPS	kafkabeat

  

  To configure the Kafka Beat for SentinelOne, see:

  SentinelOne via Kafka Beat Setup

  Configure the SentinelOne Log Source in SIEM

Initialize the Beat 

1. Confirm Open Collector is running:

   CODE

   ./lrctl status

   You should see the open_collector and metrics versions:

   

   

   If Open Collector is not running correctly, see Troubleshoot the Open Collector in the Open Collector Installation and User Guide.

2. Start the Beat:

   CODE

   ./lrctl kafkabeat start

3. Select New kafkabeat instance from the list and hit Enter.
   

   

4. Enter the unique beat identifier for this Kafka Beat instance and hit Enter.
   

   

   
   

5. Enter the Kafka broker as: KAFKA_SERVER_HOST:KAFKA_SERVER_PORT, replacing KAFKA_SERVER_HOST with the IP of the Kafka server, and KAKFA_SERVER_PORT with the server port Kafka is listening to.
   

   

   

   Alternatively, you can use a fully-qualified domain name (FQDN) as the KAFKA_SERVER_HOST as shown below: 

   

6. Enter the Kafka topic name from which you want to collect data. It should be the same topic on which your producer is sending data.
   

   

7. Enter the consumer ID for this beat instance. You can always change the consumer ID by using config edit to fetch messages from the beginning.
   

   

8. Select one of the supported authentication mechanisms from the list that your Kafka server supports.
   

   

9. Enter Y to skip host verification on the Kafka Beat. This is recommended for self-signed certificates.

   

   If you enter N, then you have to upload the certificate file in the following steps. Otherwise, skip to step 10.

   

   1. Paste your server SSL certificate. (All content, including '-----------BEGIN CERTIFICATE------------------' to '-----------------------END CERTIFICATE------------------'). 
      

      

   2. Paste your client SSL certificate. (All content, including '-----BEGIN CERTIFICATE-----' and '-----END CERTIFICATE-----').
      

      

      

      If your certificate files have more than one BEGIN CERTIFICATE line, paste the section that contains a friendly name localhost or your actual hostname. This is usually the section in the middle.

   3. Paste your private key file content. (All content, including '-----BEGIN PRIVATE KEY-----' and '-----END PRIVATE KEY-----').

      

10. Complete the following steps only if the SASL_SSL authentication mechanism was selected in step 8. Otherwise, skip to step 11.

    1. Select the SASL_SSL mechanism you want to enable in the Kafka Beat.

       

    2. Enter the user name to use for SASL authentication.
       

       

    3. Enter the password for SASL authentication.
       

       

11. Enter Y if you want to enable SentinelOne parsing support within the Kafka Beat.

    

     The Kafka Beat starts.

12. Check the status of the service:

    CODE

    ./lrctl kafkabeat status

    

Default Config Values for Kafka Beat

S.No	Field Name	Default Values
1	heartbeatinterval	60s
2	heartbeatdisabled	false
3	broker	User Provided Combination of the broker's IP or FQDN and broker's port in the format IP:Port.
4	topic	User Provided Note: Valid characters for the topic name are "[a-z, A-Z, 0-9, and \\._\\-]". Using any other characters will result in a error.
5	tls_enabled	true Enables or disables SSL support on the Kafka Beat.
6	tls_enabled_skip_hostverify	false Enables or disables host verification in SSL Mode (recommended for self-signed certificates).
7	consumerID	User Provided This must be unique for each beat instance. The user can change the consumer ID whenever the logs needed to be fetched from the beginning. Valid characters for the consumer ID are "[a-z, A-Z, 0-9, and \\._\\-]". Using any other characters will result in a error.
8	consumerWaitTime	500 The maximum amount of time, in milliseconds, the broker waits for consumer acknowledgment before declaring it to be disconnected.
9	maxProcessingTime	5000 The maximum amount of time, in milliseconds, the consumer expects a message to take to process for the user.
10	sasl_enabled	false Note Enables or disables SASL support in the Kafka Beat.
11	sasl_mechanism	SCRAM_512 Note The SASL mechanism to be used by the Kafka Beat. This should always be same as the Kafka server SASL mechanism. The following are the SASL mechanisms currently supported by the Kafka Beat. PLAIN: For no encryption SCRAM_256: for SHA256 encryption SCRAM_512: for SHA512 encryption
12	username	User Provided Username for SASL authentication on the Kafka server.
13	password	User Provided Password for SASL authentication on the Kafka server.
14	sentinel1ParsingEnable	false Enables or disables SentinelOne message parsing, allowing SentinelOne to send compressed protocol buffer (protobuf) messages.