Skip to main content
Skip table of contents

Create Custom Open Collector Pipelines

The following procedure describes how to install the Open Collector Pipeline and LRJQ utilities to configure a custom log source. This guide uses WinLogBeat as an example. 

LogRhythm Support can only assist with basic installation and configuration of the Open Collector Pipeline and LRJQ utilities. For assistance with custom parsing, you need to work with Professional Services.

  1. Download the latest lrctl script from the LogRhythm Community.

    If you have not already installed the Open Collector, follow the instructions in the Open Collector Installation and User Guide, and then return to this guide.

  2. In your Open Collector host, run the following command to ensure you have the latest Open Collector.

    CODE
    ./lrctl oc restart
  3. Install the ocpipeline tool.

    CODE
    ./lrctl ocpipeline install
  4. Install the lrjq utility.

    CODE
    ./lrctl lrjq install
  5. Create the new pipeline.

    CODE
    ./ocpipeline create --name winlogbeat --destination .

    The new winlogbeat directory appears.

    -rwxrwxr-x. 1 logrhythm logrhythm 17643 Feb  7 11:56 lrctl
    -rw-rw-r--. 1 logrhythm logrhythm  2331 Dec 10 15:28 lrctl.json
    -rwxr-xr-x. 1 logrhythm logrhythm  2390 Feb 10 11:20 lrjq
    -rwxr-xr-x. 1 logrhythm logrhythm  1637 Feb  7 11:57 ocpipeline
    drwxr-xr-x. 3 logrhythm logrhythm   125 Feb 10 14:31 winlogbeat
    -rw-r--r--. 1 logrhythm logrhythm  1748 Feb 10 16:42 winlogbeat.pipe
    -rw-rw-r--. 1 logrhythm logrhythm  2443 Feb 10 11:04 winlogbeat_sample.json
  6. Open the is_winlogbeat.jq filter.

    CODE
    vi winlogbeat/is_winlogbeat.jq

    The is_winlogbeat.jq filter checks whether the data matches the winlogbeat criteria.

  7. Edit the is_winlogbeat.jq filter according to the information below.

    def is_winlogbeat:
        # TODO: fill out with logic for pipeline include (INTENTIONALLY DOES NOT PASS AS IS)
        ."@metadata".beat == "winlogbeat"
    ;
  8. To test the filter, create a sample file with a single log entry for the log source you are configuring and give it a name, in this case winlogbeat_sample.json, and then run the following command:

    CODE
    cat winlogbeat_sample.json | ./lrjq run -f winlogbeat/include.jq

    The output.file functionality can help you configure this file. For more information, see Step 7 in Configure a Device for a Custom OC Pipeline.

  9. Open the winlogbeat.jq transform.

    CODE
    sudo vi winlogbeat.jq
  10. Edit the winlogbeat.jq transform according to the information below.

    # get_io_format converts incoming data to a standard IO format. The original
    # content is added to the output in the "original_message" field automatically.
    def get_io_format:
    {
      "input": .,
      "output": {
        "original_message": . | tojson
      }
    }
    # transform will normalize the incoming log into the LogRhythm Schema
    # that can then be forwarded to the SIEM
    def transform:
      get_io_format |
      .output.beatname    = .input."@metadata".beat                      |
      .output.sname       = .input.winlog.computer_name                  |
      .output.vmid        = .input.winlog.event_id                       |
      .output.sport       = .input.winlog.event_data.SourcePort          |
      .output.sip         = .input.winlog.event_data.SourceAddress       |
      .output.protnum     = .input.winlog.event_data.Protocol            |
      .output.process     = .input.winlog.event_data.Application         |
      .output
    ;
  11. Test the transform using the sample file created in Step 8.

    CODE
    cat winlogbeat_sample.json | ./lrjq run -f winlogbeat/transform.jq
  12. Test the pipeline in its entirety using the sample file created in Step 8.

    CODE
    cat winlogbeat_sample.json | ./ocpipeline test --source winlogbeat/
  13. Package and import the pipeline.

    CODE
    ./ocpipeline package --source winlogbeat --destination . --force

    If you have an existing winlogbeat.pipe file in your directory, the force flag is required to overwrite it.

  14. Import the .pipe file into your Open Collector.

    CODE
    cat winlogbeat.pipe | ./lrctl oc pipe import
  15. Restart Services.

    CODE
    ./lrctl metrics restart
    ./lrctl oc restart
  16. Validate the status of all pipelines.

    CODE
    ./lrctl oc -- pipe status
    Pipeline Name
    winlogbeat/winlogbeat
    logrhythm/gmail_message_tracking/gmail_message_tracking_pipe
    logrhythm/sophos_central/sophos_central_pipe
    logrhythm/pubsub/pubsub_pipe
    logrhythm/s3/s3_pipe
    logrhythm/eventhub/eventhub_pipe
    logrhythm/gsuite/gsuite_pipe
    logrhythm/heartbeat/heartbeat_pipe
    logrhythm/generic/generic_pipe
    Enabled
    yes yes yes yes yes yes yes yes yes

    The output will show a list of existing pipelines and their corresponding status.  An example can be found below:

After you complete this procedure, you will need to take additional steps to configure both the LogRhythm SIEM and the device on which the custom Beat is running.

JavaScript errors detected

Please note, these errors can depend on your browser setup.

If this problem persists, please contact our support.