Create Custom Open Collector Pipelines

The following procedure describes how to install the Open Collector Pipeline and LRJQ utilities to configure a custom log source. This guide uses WinLogBeat as an example. 

LogRhythm Support can only assist with basic installation and configuration of the Open Collector Pipeline and LRJQ utilities. For assistance with custom parsing, you need to work with Professional Services.

1. Download the latest lrctl script from the LogRhythm Community.
   

   

   If you have not already installed the Open Collector, follow the instructions in the Open Collector Installation and User Guide, and then return to this guide.

2. In your Open Collector host, run the following command to ensure you have the latest Open Collector.

   CODE

   ./lrctl oc restart

3. Install the ocpipeline tool.
   

   CODE

   ./lrctl ocpipeline install

4. Install the lrjq utility.
   

   CODE

   ./lrctl lrjq install

5. Create the new pipeline.
   

   CODE

   ./ocpipeline create --name winlogbeat --destination .

   The new winlogbeat directory appears.

   -rwxrwxr-x. 1 logrhythm logrhythm 17643 Feb  7 11:56 lrctl -rw-rw-r--. 1 logrhythm logrhythm  2331 Dec 10 15:28 lrctl.json -rwxr-xr-x. 1 logrhythm logrhythm  2390 Feb 10 11:20 lrjq -rwxr-xr-x. 1 logrhythm logrhythm  1637 Feb  7 11:57 ocpipeline drwxr-xr-x. 3 logrhythm logrhythm   125 Feb 10 14:31 winlogbeat -rw-r--r--. 1 logrhythm logrhythm  1748 Feb 10 16:42 winlogbeat.pipe -rw-rw-r--. 1 logrhythm logrhythm  2443 Feb 10 11:04 winlogbeat_sample.json

6. Open the is_winlogbeat.jq filter.
   

   CODE

   vi winlogbeat/is_winlogbeat.jq

   

   The is_winlogbeat.jq filter checks whether the data matches the winlogbeat criteria.

7. Edit the is_winlogbeat.jq filter according to the information below.

   def is_winlogbeat:     # TODO: fill out with logic for pipeline include (INTENTIONALLY DOES NOT PASS AS IS)     ."@metadata".beat == "winlogbeat" ;

8. To test the filter, create a sample file with a single log entry for the log source you are configuring and give it a name, in this case winlogbeat_sample.json, and then run the following command:

   CODE

   cat winlogbeat_sample.json | ./lrjq run -f winlogbeat/include.jq

   

   The output.file functionality can help you configure this file. For more information, see Step 7 in Configure a Device for a Custom OC Pipeline.

9. Open the winlogbeat.jq transform.

   CODE

   sudo vi winlogbeat.jq

10. Edit the winlogbeat.jq transform according to the information below.

    # get_io_format converts incoming data to a standard IO format. The original # content is added to the output in the "original_message" field automatically. def get_io_format: {   "input": .,   "output": {     "original_message": . | tojson   } } ;  # transform will normalize the incoming log into the LogRhythm Schema # that can then be forwarded to the SIEM def transform:   get_io_format |   .output.beatname    = .input."@metadata".beat                      |   .output.sname       = .input.winlog.computer_name                  |   .output.vmid        = .input.winlog.event_id                       |   .output.sport       = .input.winlog.event_data.SourcePort          |   .output.sip         = .input.winlog.event_data.SourceAddress       |   .output.protnum     = .input.winlog.event_data.Protocol            |   .output.process     = .input.winlog.event_data.Application         |   .output ;

11. Test the transform using the sample file created in Step 8.
    

    CODE

    cat winlogbeat_sample.json | ./lrjq run -f winlogbeat/transform.jq

12. Test the pipeline in its entirety using the sample file created in Step 8.

    CODE

    cat winlogbeat_sample.json | ./ocpipeline test --source winlogbeat/

13. Package and import the pipeline.
    

    CODE

    ./ocpipeline package --source winlogbeat --destination . --force

    

    If you have an existing winlogbeat.pipe file in your directory, the force flag is required to overwrite it.

14. Import the .pipe file into your Open Collector.
    

    CODE

    cat winlogbeat.pipe | ./lrctl oc pipe import

15. Restart Services.

    CODE

    ./lrctl metrics restart
    ./lrctl oc restart

16. Validate the status of all pipelines.

    CODE

    ./lrctl oc -- pipe status

    Pipeline Name winlogbeat/winlogbeat logrhythm/gmail_message_tracking/gmail_message_tracking_pipe logrhythm/sophos_central/sophos_central_pipe logrhythm/pubsub/pubsub_pipe logrhythm/s3/s3_pipe logrhythm/eventhub/eventhub_pipe logrhythm/gsuite/gsuite_pipe logrhythm/heartbeat/heartbeat_pipe logrhythm/generic/generic_pipe	Enabled yes yes yes yes yes yes yes yes yes

    The output will show a list of existing pipelines and their corresponding status.  An example can be found below:

After you complete this procedure, you will need to take additional steps to configure both the LogRhythm SIEM and the device on which the custom Beat is running.