Create Custom Open Collector Pipelines
The following procedure describes how to install the Open Collector Pipeline and LRJQ utilities to configure a custom log source. This guide uses WinLogBeat as an example.
LogRhythm Support can only assist with basic installation and configuration of the Open Collector Pipeline and LRJQ utilities. For assistance with custom parsing, you need to work with Professional Services.
Download the latest lrctl script from the LogRhythm Community.
If you have not already installed the Open Collector, follow the instructions in the Open Collector Installation and User Guide, and then return to this guide.
In your Open Collector host, run the following command to ensure you have the latest Open Collector.
CODE./lrctl oc restart
Install the ocpipeline tool.
CODE./lrctl ocpipeline install
Install the lrjq utility.
CODE./lrctl lrjq install
Create the new pipeline.
CODE./ocpipeline create --name winlogbeat --destination .
The new winlogbeat directory appears.
-rwxrwxr-x. 1 logrhythm logrhythm 17643 Feb 7 11:56 lrctl
-rw-rw-r--. 1 logrhythm logrhythm 2331 Dec 10 15:28 lrctl.json
-rwxr-xr-x. 1 logrhythm logrhythm 2390 Feb 10 11:20 lrjq
-rwxr-xr-x. 1 logrhythm logrhythm 1637 Feb 7 11:57 ocpipeline
drwxr-xr-x. 3 logrhythm logrhythm 125 Feb 10 14:31 winlogbeat
-rw-r--r--. 1 logrhythm logrhythm 1748 Feb 10 16:42 winlogbeat.pipe
-rw-rw-r--. 1 logrhythm logrhythm 2443 Feb 10 11:04 winlogbeat_sample.jsonOpen the is_winlogbeat.jq filter.
CODEvi winlogbeat/is_winlogbeat.jq
The is_winlogbeat.jq filter checks whether the data matches the winlogbeat criteria.
Edit the is_winlogbeat.jq filter according to the information below.
def is_winlogbeat:
# TODO: fill out with logic for pipeline include (INTENTIONALLY DOES NOT PASS AS IS)
."@metadata".beat == "winlogbeat"
;
To test the filter, create a sample file with a single log entry for the log source you are configuring and give it a name, in this case winlogbeat_sample.json, and then run the following command:
CODEcat winlogbeat_sample.json | ./lrjq run -f winlogbeat/include.jq
The output.file functionality can help you configure this file. For more information, see Step 7 in Configure a Device for a Custom OC Pipeline.
Open the winlogbeat.jq transform.
CODEsudo vi winlogbeat.jq
Edit the winlogbeat.jq transform according to the information below.
# get_io_format converts incoming data to a standard IO format. The original
# content is added to the output in the "original_message" field automatically.
def get_io_format:
{
"input": .,
"output": {
"original_message": . | tojson
}
}
;# transform will normalize the incoming log into the LogRhythm Schema
# that can then be forwarded to the SIEM
def transform:
get_io_format |
.output.beatname = .input."@metadata".beat |
.output.sname = .input.winlog.computer_name |
.output.vmid = .input.winlog.event_id |
.output.sport = .input.winlog.event_data.SourcePort |
.output.sip = .input.winlog.event_data.SourceAddress |
.output.protnum = .input.winlog.event_data.Protocol |
.output.process = .input.winlog.event_data.Application |
.output
;Test the transform using the sample file created in Step 8.
CODEcat winlogbeat_sample.json | ./lrjq run -f winlogbeat/transform.jq
Test the pipeline in its entirety using the sample file created in Step 8.
CODEcat winlogbeat_sample.json | ./ocpipeline test --source winlogbeat/
Package and import the pipeline.
CODE./ocpipeline package --source winlogbeat --destination . --force
If you have an existing winlogbeat.pipe file in your directory, the force flag is required to overwrite it.
Import the .pipe file into your Open Collector.
CODEcat winlogbeat.pipe | ./lrctl oc pipe import
Restart Services.
CODE./lrctl metrics restart ./lrctl oc restart
Validate the status of all pipelines.
CODE./lrctl oc -- pipe status
Pipeline Name
winlogbeat/winlogbeat
logrhythm/gmail_message_tracking/gmail_message_tracking_pipe
logrhythm/sophos_central/sophos_central_pipe
logrhythm/pubsub/pubsub_pipe
logrhythm/s3/s3_pipe
logrhythm/eventhub/eventhub_pipe
logrhythm/gsuite/gsuite_pipe
logrhythm/heartbeat/heartbeat_pipe
logrhythm/generic/generic_pipeEnabled
yes yes yes yes yes yes yes yes yesThe output will show a list of existing pipelines and their corresponding status. An example can be found below:
After you complete this procedure, you will need to take additional steps to configure both the LogRhythm SIEM and the device on which the custom Beat is running.