Generic Beat

Version 6.2.0

The Generic Beat is designed to collect data from log sources supporting similar authentication and pagination styles implemented using other beats, including authentication and authorization mechanisms. Users can configure any log source beat by providing log source credentials during the beat's configuration, provided that it follows the same API standard as supported by the Generic Beat.

Components of a REST API Request/Response

Implementing the Generic Beat requires an understanding of REST API requests/responses. A REST API request/response pair can be separated into six components:

1. The request URI, which consists of: 

   CODE

   {URI-scheme} :// {URI-host} / {resource-path} ? {query-string}

   Although the request URI is included in the request message header, it is listed separately because most languages or frameworks require it to be passed separately from the request message.

   • URI-scheme. Indicates the protocol used to transmit the request (for example, http or https).
   • URI-host. Specifies the domain name or IP address of the server where the REST service endpoint is hosted (for example, graph.microsoft.com).
   • Resource-path. Specifies the resource or resource collection, which may include multiple segments used by the service in determining the selection of those resources.
     For example, beta/applications/00003f25-7e1f-4278-9488-efc7bac53c4a/owners can be used to query the list of a specific application's owners within the applications collection.
   • (Optional) Query-string. Provides additional simple parameters, such as the API version or resource selection criteria.
2. HTTP request message header fields:
   
   • A required HTTP method (also known as an operation or verb), which tells the service what type of operation is being requested. For more information, see HTTP Method Definitions.
   • Optional additional header fields, as required by the specified URI and HTTP method.
     For example, an authorization header that provides a bearer token containing client authorization information for the request.

3. (Optional) HTTP request message body fields to support the URI and HTTP operation.
   The current version of the Generic beat supports the GET method, and has limited support for the POST method.

   

   The POST method is supported for APIs having header-based authentication without any filters or sorting, and its request body should only have one key:value pair.

4. HTTP response message header fields:
   
   • An HTTP status code, ranging from 2xx success codes to 4xx or 5xx error codes. For more information, see HTTP Status Codes. Alternatively, a service-defined status code may be returned, as indicated in the API documentation.
   • Optional additional header fields as required to support the request's response, such as a content-type response header.
5. (Optional) HTTP response message body fields. 
   MIME-encoded response objects are returned in the HTTP response body, such as a response from a GET method that is returning data. Typically, these objects are returned in a structured format such as JSON or XML, as indicated by the content-type response header.

6. HTTP request method: GET or POST.

   

   The POST method is supported for APIs having header-based authentication without any filters or sorting, and its request body should only have one key:value pair.

Generic Beat Overview

The following API components are necessary to configure the Generic Beat:

Request URI

This assumes that the user is aware of the API endpoint behavior and provides the complete request URI, including URI scheme, URI host, and resource path.

Query Parameter

Any number of query string parameters can be provided in the key:value format as an input in the lrctl query parameter prompt.

HTTP Request Message Header Fields

Custom headers are supported in the Generic Beat's current version. Any number of custom headers can be provided in the key:value format as a input in lrctl header prompt.

HTTP Request Method

The GET and POST request methods are supported.

The POST method is supported for APIs having header-based authentication without any filters or sorting, and its request body should only have one key:value pair.

(Optional) HTTP Request Message Body Fields

Since the GET and POST methods are currently supported in the Generic Beat, the request body field support is optional.

The POST method is supported for APIs having header-based authentication without any filters or sorting, and its request body should only have one key:value pair.

Authentication/Authorization

The Generic Beat supports the following authentication mechanisms: 

• Basic authentication (username/password-based).

• Header-based authentication.

• OAuth2.0 authentication mechanism.

  

  OAuth 2 authentication has two token types: Access Token and Refresh Token. As of the 2021.10 release (Generic Beat version 6.0.0), the Access Token is the only supported token type. The Refresh Token is not currently supported.

Pagination, Filtering, and Sorting

• Limit-offset based, page number-based, and cursor-based pagination.

• Sorting can be implemented by adding the query parameter in the request parameter field.

• Filtering can be achieved by adding the query parameter in the request parameter field or by using the existing filters in the Generic Beat.

API-Specific Components of the Generic Beat

When configuring the Generic Beat, you will need to use the authentication/authorization method, pagination method, and filtering method that your API accepts. Additionally, a sorting method can be configured if one is accepted by your API.

Authentication/Authorization Methods

Authentication means confirming your own identity, while authorization means granting access to the system. In simple terms, authentication is the process of verifying who you are, while authorization is the process of verifying what you have access to.

Basic Authentication

This is the most straightforward and easiest method. With this method, the sender places a username:password into the request header. The username and password are encoded with Base64, which is an encoding technique that converts the username and password into a set of 64 characters to ensure safe transmission. A basic authentication in a request header will look similar to the following:

Header-based Authentication

Header-based authentication (also called token authentication) is an HTTP authentication scheme that involves security tokens called bearer tokens. The client must send this token in the authorization header when making requests to protected resources.

OAuth (2.0) Access Token

The Access Token is sent like an API key and allows the application to access a user’s data. Access tokens can also be configured to expire.

OAuth 2 authentication has two token types: Access Token and Refresh Token. As of the 2021.10 release (Generic Beat version 6.0.0), the Access Token is the only supported token type. The Refresh Token is not currently supported.

Pagination

Most endpoints that return a list of entities will need to have some sort of pagination. Without pagination, a simple search could return millions or even billions of hits, causing extraneous network traffic. Pagination requires an implied ordering. By default, this may be the item's unique identifier but can be other ordered fields (for example, a created date).

Offset Pagination

Limit/Offset pagination can be retrieved using the following command: 

GET /items?limit=20&offset=100

This query would return 20 rows, starting with the 100th row.

Page Number-based Pagination

This is the simplest and most common form of pagination, particularly for apps that use SQL databases. Using this method, the set is divided into pages. The endpoint accepts a page parameter that is an integer indicating the page within the list to be returned.

http://lrtestapi.azurewebsites.net/countries/?starttime=2021-08-03+17%3A26%3A44.223638&endtime=2021-08-03+17%3A35%3A30.692350&page=2&pagesize=15

Cursor-based Pagination 

In cursor-based pagination, when a request is made, the server returns the first page of data and the cursor points to the next page. A cursor can be a URL to the next page or it could be a token after which the next records should be fetched.

Filtering

Between Start and End Date

This filter fetches the logs within the given range of start date and end date. Ensure the API supports this filter on the server end.
The following is an example request using the between start and end date filter.

After Any Specific Start Date

This filter takes a start value provided by the user and requests records from the server, fetching logs from after the specified start date. 
The following is an example of the after any specific start date filter.

Within an Interval

Similar to the between start and end date filter, this filter fetches the logs between any specific start time and end time, with the only difference being the format in which the start and end date are sent. In this filter, the start and end dates are sent in a single string separated by a delimiter.

The following is an example of the within an interval filter.

Sorting

Like filtering, sorting is an important feature for any API endpoint that returns a lot of data. When returning a list of users, your API users may want to sort by last modified date or by email. To enable sorting, many APIs add a sort or sort_by URL parameter that can take a field name as the value. Good API designs give the flexibility to specify ascending or descending order. Like filters, specifying the order requires encoding three components into a key/value pair.

The following are example formats for using the sorting feature:

GET /users?sort_by=asc(email)
and
GET /users?sort_by=desc(email)

GET /users?sort_by=+email
and
GET /users?sort_by=-email

GET /users?sort_by=email.asc
and
GET /users?sort_by=email.desc

GET /users?sort_by=email&order_by=asc
and
GET /users?sort_by=email&order_by=desc