Breadcrumbs

Initialize the Generic Beat with Cursor-Based Pagination, Basic Authentication, and the "After Any Specific Date" Filter

This topic outlines the process to set up the Generic beat to fetch logs from any log sources that support cursor-based pagination, basic authentication, and the "after any specific date" filter.

This guide assumes a working knowledge of the API request and response format to be set up with the Generic beat.

Prerequisites

  • Open Collector is installed. If you have not already installed it, follow the instructions in the Open Collector Installation and User Guide, and then return to this topic.

  • Log source Name (The same as the name of the log source you want to setup with the Generic beat).

  • Log source URL (The complete URL with which logs can be fetched from the log source).

  • Log source Username and Password (for basic authentication).

  • Cursor Type (for example, URL or query parameter).

  • Cursor query parameter (the fieldname in which the cursor is to be sent if the cursor type is query parameter).

  • Cursor location (for example, response body or response header).

  • Cursor field to be retrieved from the response body or response header.

  • Start date field supported by the API.

  • Field to fetch the next start date from response.

  • Time format supported by the API.

  • Time range supported by the API.

  • Number of back days data supported by the API.

  • Sorting fields, if sorting is supported by the API.

  • Response data field, if the response needs to be fetched from a specific field in JSON.

  • Period during which logs need to be fetched from the log source.

  • Request headers and query parameters that the API requires to fetch logs from an endpoint. 

  • The following port is open:

    Direction

    Port

    Protocol

    Source

    Outbound

    443/80

    HTTPS/HTTP

    genericbeat


Initialize the Beat

For more information on any of the Generic beat-specific fields described in this topic, see the Guide to Generic Beat Prompt Inputs section of the Configure the Generic Beat topic.

  1. To confirm the Open Collector is running, run the following command:

    ./lrctl status

    If the Open Collector is not running correctly, see Troubleshoot the Open Collector in the Open Collector Installation and User Guide.
    You should see the open_collector and metrics versions.
    lrctl_status.PNG

  2. To start the Beat, run the following command:

    ./lrctl genericbeat start

    genericbeat_start.PNG

    The values shown in the images used in this guide are example values. Replace the example values with the actual values that your API supports.


  3. Select New genericbeat instance using the arrow keys. Enter the unique beat identifier for this beat instance, and then press Enter.

    unique_id.PNG

  4. Enter the log source name for the Generic beat's configuration, and then press Enter.
    logsource.PNG

  5. Select GET as the HTTP request method using the arrow keys, and then press Enter.
    image2022-1-24_17-18-17.png

  6. Enter the API URL to be used for the Generic beat's configuration, and then press Enter.
    url.PNG

  7. Choose the Cursor Pagination style using the arrow keys, and then press Enter.
    cr_paginationStyle.PNG

  8. Select the cursor type supported by the API using the arrow keys, and then press Enter.
    cr_cursorType.PNG

  9. (Optional) If Query Parameter was selected as the cursor type in step 7, enter the cursor query parameter to be sent in the request.
    cr_query_param.PNG

  10. Select the location where the cursor field will come in the response using the arrow keys, and then press Enter.
    cr_headerLoc.PNG

  11. (Optional) If Response Header was selected as the cursor field location in step 9:

    1. Select the cursor header type using the arrow keys, and then press Enter.

      cr_headerType1.PNG

      1. If the cursor header type is Link, enter the link parameter to be fetched from the response link header.
        cr_linkParam.PNG

      2. If the cursor header type is Custom Header, enter the cursor header field.
        cr_headerField.PNG

  12. (Optional) If Response Body was selected as the cursor field location in step 9, enter the cursor field.
    cr_cursorField.PNG

  13. Select the Date Range filter type using the arrow keys, and then press Enter.
    filter_type.PNG

  14. Select one of the date-time formats for the date range filter using the arrow keys, and then press Enter.
    time_format.PNG

  15. Enter the delay time (in seconds) supported by the API for live logs. For example, 2s.
    delay_time.PNG

  16. Select the After any specific date filter using the arrow keys, and then press Enter.
    after_time_filter.PNG

  17. Enter the start field, and then press Enter.
    start_field.PNG

  18. Enter the start value, and then press Enter.
    start_value.PNG

  19. Specify the field to fetch from the response data in order to get the next start date, and then press Enter.
    next_start_date.PNG

  20. Select the Basic Authentication mechanism using the arrow keys, and then press Enter.
    auth_mechanism.PNG

  21. Enter the Username for basic authentication, and then press Enter.
    username.PNG

  22. Enter the Password for basic authentication, and then press Enter.
    password.PNG

  23. Select whether the API supports sorting using the arrow keys, and then press Enter.
    sorting_flag.PNG

  24. (Optional) Enter the sorting field, and then press Enter.
    sorting_field.PNG

  25. (Optional) Enter the sorting value, and then press Enter.
    sorting_value.PNG

  26. Enter the request headers (other than the authentication header) in the “key:value” format, and then press 'c' to continue.
    headers1.PNG
    headers2.PNG

  27. Enter the request body in the key:value format, and then press 'c' to continue.
    image2022-1-24_17-26-14.png

  28. Enter the request parameters (other than start time and end time) in the key:value format, and then press 'c' to continue.
    parameters.PNG

  29. To parse any specific field from the response (for example, if the API response contains logs in a specific field), select yes using the arrow keys and then press Enter.
    resp_field_flag.PNG

  30. Enter the field in the API response in which data will come (for example, "resp"), and then press Enter.
    resp_data_field.PNG

  31. Enter the polling period for the beat in seconds (for example, "30s"), and then press Enter.
    The beat is successfully initialized using cursor-based pagination, basic authentication, and the "after any specific date" filter.
    period.PNG