Configure the Generic Beat

Prerequisites

• Users should have end-to-end knowledge of the log source API to be configured. 
• The complete API URL for which data needs to be fetched.
  For example, if the API URL looks like https://api.hostname.io/<org_id>/logs, then the user should provide the URL by replacing the org_id with its actual value.
• The request parameters with their actual values in the config file.
  For example, if the parameter org_id needs to be sent as a request parameter, then the user must provide this inside the parameter section of the config file.
• The http headers that need to be sent in the request.

Guide to Generic Beat Prompt Inputs

Log Source Name

Provide the name of the log source from which the beat should pull data.

For example, to configure the Generic Beat for Proofpoint log sources, enter proofpoint here. 

API URL

Provide the API URL used to get data from the log source. The API URL consists of the API hostname and API endpoint only; do not include the query parameters string.

For example, the following image highlights the URL that should be entered when configuring the Generic Beat for Cisco AMP API audit logs. 

Pagination Styles

The Generic Beat supports four types of pagination styles, as listed below. Before configuring the Generic Beat for log sources, you should learn the types of pagination styles that your API supports using the following methods.

Cursor-based Pagination

Check if the API returns the next page URL or next token to fetch the next set of records.

If so, the API supports the cursor-based pagination style. The images below show the two different types of cursors supported by APIs: next token, i.e. "after," and next page URL, i.e. "next."

In cursor-based pagination, when a request is made, the server returns the first page of data and the cursor that points to the next page. A cursor can be a URL to the next page or it could be the token after which the next records should be fetched. The Generic Beat asks for the following inputs to configure cursor pagination:

• Cursor Type

  The cursor type can be a URL to the next set of data records or it can be a token to fetch the next set of records as a query parameter. For example, in the image below, the cursor type is URL.

  

  
  

  In this image, the cursor type is a token to the next set of records, achieved by adding it into the URL as a query parameter.

  

  
  

  

  
  
  

• Cursor Location

  The cursor can be located in response headers or the response body. In the following image, the cursor appears in the response header, in which case the cursor location should be Response Header. 
  
  

  

  
  If the cursor appears in the response body, such as in the following image, then select Response Body.

  

  
  

• Cursor Field (If cursor location is payload)

  The cursor field refers to the field name in which the cursor appears inside the response body.

• Cursor Query Param (This prompt only appears if the cursor type is query parameter)

  Cursor query param is the field name in the request query param in which the next token will be sent. In the following image, the query param is after.

  

• Cursor Header Type (If cursor location is the header)

  The cursor header type can be either a link-defined HTTP header, or a custom header that can be any named field.

• Cursor Header Field

  The cursor header field is the field name from which the token is pulled. If it is a link header, it is recommended to provide the rel value here. In the image below, the cursor header field is next.

  

Page-based Pagination

If cursor-based pagination is not valid for the API, the API may support pagination using the page number and any specific page size.
If so, the API supports page-based pagination. The image below shows the page number-based API, where "limit" defines the page size and "page" defines the page index.

http://lrtestapi.azurewebsites.net/countries/?starttime=2021-08-03+17%3A26%3A44.223638&endtime=2021-08-03+17%3A35%3A30.692350&page=2&pagesize=15

• Page size field. The field name for page size. In the above example, the page size field is pagesize.
• Page size value. The value for page size to limit the number of records returned by the server in a single request. In the above example, the page size value is 15.
• Page number field. The field name for the page number. In the above example, the page number field is page.

Limit Offset Pagination

If cursor- and page-based pagination are not valid for the API, the API may support any limit offset-based query parameter to fetch the records.

If so, the API supports limit offset-based pagination style. The image below shows an example of limit offset-based pagination, where "limit" defines the total number of records to return and "offset" defines the index after which the records have to be returned.

Limit offset-based pagination is a technique wherein the client requests parameters with a specific limit (the number of results) and offset (the number of records that need to be skipped). Offset-based pagination is easy to use and is preferred for static data. While retrieving data for offset pagination, one generally requires two parameters in the query: a limit and an offset. The image below shows an example of a limit offset-based API.

• Limit Field. The field name to be used as the limit. In the above image, the limit field is limit.

• Limit Value. The total number of records to be returned in a single request. In the above image, the limit value is 100.

• Offset Field. The field name to be used as the offset. In the above image, the offset field is offset.

No Pagination

If the above pagination styles are not valid for the API, then it doesn't support any pagination style compatible with the Generic Beat. In this case, select no pagination from the pagination dropdown.

Filter Styles

Date Range 

When choosing the Date Range filter, the following two prompts also appear:

• Time format. This refers to the date-time format that the API supports, i.e. RFC3339 or ISO8601.
  Select the appropriate time format from the drop-down menu: Please choose one of the date-time formats for the date range filter.
• Delay time. Some APIs don't support exact real-time log fetching. In this case, add the delay in seconds that the API supports. For example, if the API supports a five seconds delay, enter 5s.

• Between start and end date
  This filter fetches logs within the range of a given start date and end date. The API must support this filter on the server end. The image below shows an example request using the between start and end date filter.
  
  

  

  
  

  Start field. The field name used to send the start time in the request. 

  Start value. The start date value from which logs should be fetched from the server. When giving the start date as back days, ensure the API supports that number of back days to fetch logs. For example, if the start value is 7 days before the current date, then the API must support 7 day backlogs; otherwise, the beat will return an error.

  End field. The field name to be used to send the end date in the request.

  End value. The end date value must be set according to the maximum interval that the API supports. For example, if the API supports a maximum of one hour logs in a single API request and the start value is 2021-10-02T08:00:00Z, then the end value must be 2021-10-02T09:00:00Z, for a difference of one hour.

• After any specific start date
  This filter takes a start value provided by the user and requests records from the server for any time after that date. 
  The image below shows an example of the after any specific start date filter.
  
  

  

  
  

  Start field. The field name used to send the start time in the request. For example, in the image above, the start field is since.

  Start value. The start date value from which logs should be fetched from the server. In the image above, the start value is 2017-10-01T00:00:00.000Z.

  

  When giving the start date as back days, ensure the API supports that number of back days to fetch logs. For example, if the start value is 7 days before the current date, then the API must support 7 day backlogs; otherwise, the beat will return an error. 

• Next start date response field

  This is the date field to parse from the response to fetch the next set of records. The below image shows an example of the next start date. Enter data.date as the next start date response field.

  

  
  

• Within an interval
  This filter fetches logs between any specific start time and end time similar to the between start and end date filter. However, in the within an interval filter, the start and end date are sent in a single string separated by a delimiter. The image below shows an example of the within an interval filter. 
  
  

  

  
  

  Interval field. The field name in which the date and time are sent. In the image above, the interval field is interval.

  Interval value. Interval values contain the start time and end time values separated by delimiter, as supported by the API. For example, in the image above, the interval value is 2021-10-08T00:00:00.000Z/2021-10-08T01:00:00.000Z.

  

  When giving the start date as back days, ensure the API supports that number of back days to fetch logs. For example, if the start value is 7 days before the current date, then the API must support 7 day backlogs; otherwise, the beat will return an error.

  The end date value must be set according to the maximum interval that the API supports. For example, if the API supports a maximum of one hour logs in a single API request and the start value is 2021-10-02T08:00:00Z, then the end value must be 2021-10-02T09:00:00Z, for a difference of one hour.

  Split char. The delimiter between the start and end time in the interval value. For example, in the image above, the delimiter is /.

No Filter 

If the API doesn't support any of the filters above, then select no filter.

Authentication Types

The Generic Beat supports three types of authentication mechanisms. Select one of the authentication types below that is supported by the API in order to configure the Generic Beat.

Basic

Basic authentication requires a username and password to authenticate the request on the server.

• Username. The username required for basic authentication. Enter the authentication username in the prompt: Please enter Username for authentication.
• Password. The password required for basic authentication. Enter the authentication password in the prompt: Please enter Password for authentication.

Header-based

In header-based authentication, the client can send an authorization header to be accepted by the server when it makes its original request, potentially avoiding the challenge and response process.

• Auth header field. The authentication header field needed to be sent in the request. For example, Authorization.
• Auth token value. The token value sent in the request, including the append field, like Bearer or SSWS. For example, SSWS 00uH4WDqFWdAlVNDzXQOLQBEA-JtlNzgCV4TwB.

OAuth2.0 

OAuth is an open-standard authorization protocol or framework that describes how unrelated servers and services can safely allow authenticated access to their assets without actually sharing the initial, related, single login credential. In authentication parlance, this is known as secure, third-party, user-agent, delegated authorization.

OAuth2.0 authentication has two token types: Access Token and Refresh Token. As of the 2021.10 release (Generic Beat version 6.0.0), the Access Token is the only supported token type. The Refresh Token is not currently supported.

• Provider URL. The URL of the authentication server. For example, https://gateway.qg1.apps.qualys.in/auth.

• Request method. The request method to get the access_token from the authentication server. This could be GET or POST.

• Content type. The content type of the request body. This prompt will only appear if the request method is POST. Choose one of the following content types as supported by the API. Content types supported by the Generic Beat are:

• • application/json
  • application/x-www-form-urlencoded

• (Optional) Request body. This prompt only appears if the content-type is application/json. Provide the entire JSON body to receive the access token from the server. The image below shows an example response body to request an access token.

  

• (Optional) Request body param. This prompt only displays if the content type is application/x-www-form-urlencoded. Provide the body parameters in the key:value format. For example, to send the username in the body parameter, enter username:<YOUR_USERNAME>. The image below shows an example of the application/x-www-form-urlencoded request.

  

  
  

• OAuth request headers. The request headers supported by the API can be provided here. Headers must be entered in the key:value format. Enter c when all headers have been entered. For example, to send application/json in the request, enter Content-Type:application/json in the prompt: Please provide the headers in key:value format that is needed to be sent in the request to get access token or press c to continue.
  

• OAuth request parameters. The query parameters supported by the API can be provided here. Parameters must be entered in the key:value format. Enter "c" when all parameters have been entered. For example, to send query param limit=100 in the request, enter limit:100 in the prompt: Please provide the parameters in key:value format that is needed to be sent in the request to get access token or press c to continue.

• Access token format. The Generic Beat supports two types of access token format. Select the option supported by the API.

  • Normal text format: Contains the access token in the text/plain;charset=UTF-8 form, as shown in the image below.

    

  • JSON Format: Contains the access token in JSON body, as shown in the image below.

    

• Access token field. This prompt only appears if the access token format is JSON. Enter the field name in which the access token is coming in the response. In the JSON format image above, the access token field is access_token.

• Authorization header field. The authorization header used to send the access token from auth server, in order to get the data records from the log source server. For example, in the image below, the authorization header field is authorization.

  

• Auth token append field. The field to append in the retrieved access token when requesting data from the log source server. For example, in the image above, the auth token field is Bearer.

No Authentication 

If the API doesn't support any authentication method, select No authentication in the authentication types prompt.

Sorting Fields

If the API supports sorting, select Yes in the prompt: Does your API support sorting?

• Sorting Field. The field name that the API supports for sorting. For example, in the image below the sorting field is sortOrder.

• Sorting Value. The value used to determine sorting order. For example, in the image below the sorting value is ASCENDING.

  

Request Headers

In this field, provide the request headers that the API supports. Headers must be entered in the key:value format. Enter c when all headers have been entered. For example, to send Content-Type=application/json in the request, enter Content-Type:application/json in the prompt: Please enter the request headers other than authentication header, in “key:value” format. Press 'c' to continue.

Request Body

In this field, provide the request body that the API supports. The body must be entered in the key:value format. Enter c when the body has been entered. For example, to send limit:5 in the request, enter limit:5 in the prompt. Press 'c' to continue.

In the current version of Generic Beat, only one key:value pair is supported. This means the request body can't have multiple key:value pairs for making POST requests.

Request Parameters

Response Data Fields

Some servers send data records in a specific field in JSON response. In the first example image below, the response structure only contains the data records that are needed to parse on Open Collector. In the second example image below, the response structure contains data records inside the "data" field. The response data field must be provided to the Generic Beat in the prompt: Do you want to parse any specific field from response. This ensures the beat will send only useful data to Open Collector.

• Response data field. Refers to the field name that contains the data records. In the image above, the response data field is data.

Period

The period value defines the polling interval to fetch records from the server, in seconds. If the API server has a request limit, set the period value accordingly to avoid the "request limit error" with error code 429. For example, if the API supports 1800 requests per 24 hours, then set the period value to be 60s.