Configure the Generic REST Collector in Axon

The section details the steps required to configure a generic REST collector within Axon.

Add a Generic REST Collector

Only Administrators can take this action.

In the lower-left corner of the main screen, click the Administration cog.
The Administration menu appears on the left side.
Under Integrations, click Collectors.
The Collectors page appears.
In the upper-right corner, click + Add Collector.
The Add Collector Wizard appears.
Select Generic Rest Collector from the list of collector types.
A list of Log Source Types associated with the Generic Rest Collector appears on the right.
Click Configure.

The Generic REST Collector has four separate tabs for configuration, all of which can be navigated through freely at any time. For the best results, follow the sections below in order.
Once all tabs have been completed, advance to the Validate the Configuration section.

Collection Tab

By default, the Generic REST Collector configuration opens on the Collection tab.

Enter the following information on the configuration page's Collection tab.

Field	Description
Collector Name	Enter a unique name for this collector.
Description	Enter an optional detailed description for this collector.
Collection Schedule	Select one of the collection schedule options below, and then configure how often the collector sends new requests.
	Interval	Enter a value and select a time interval (Minutes, Hours, Days) from the drop-list.
	Cron Expression	Enter a Cron expression to determine how often logs are collected. The default, 0 /10 ? * *, allows for collection every ten minutes. Example Cron expressions and a Cron expression generator can be found at https://www.freeformatter.com/cron-expression-generator-quartz.html
Collection Period	Enter values into the provided fields to determine the duration of each log collection.
Backfill Period for Collection	Enter values into the provided fields to determine the start time when this collector runs for the first time. For example, entering 30 into the mins field collects logs from the previous 30 minutes the first time this collector runs.
Date Format	Select one of the following options to determine the date and time format supported by the log source API.
	UTC Standard	Select if time is specified in the standard date format. https://XXXXXXXX/all?format=json&interval=2023-03-18T08:16:00.000Z/2023-03-18T08:20:00.000Z In the example above, time is specified in the UTC Standard format (bolded).
	Epoch Milliseconds (Unix Time)	Select if time is specified in the Epoch or Unix format. https://XXXXXXXX/modelbreaches?starttime=1653547222000&endtime=1675266783000 In the example above, time is specified in the Epoch or Unix format (bolded).
Collection Timeout	Enter the period after which collection will time out while stalled. The default of 300 seconds means that after 300 seconds of failing to collect, the collection will time out.

Request Tab

On the configuration page, click the Request tab to configure the information below.

Enter the following information on the configuration page's Request tab. Required fields are marked with an asterisk *.

Field	Description
Endpoint *	Select either GET or POST from the drop-list, and then enter the API URL used to retrieve data from a log source. This URL includes the hostname, endpoint, and URL parameters.
Header Parameters	Enter request header parameters supported by the API as Name:Value pairs. The default header parameters Content-Type and Accept are shown in the example below. Click New to add a new row for parameters, or click the X to remove a row.
Query Parameters	Enter query parameters supported by the API as Name:Value pairs. Parameters for which the value remains the same throughout all requests are called static parameters; for example, limit. Parameters for which the value changes in an API request and can be managed by the system are called dynamic parameters; for example, start_time. For information on static and dynamic parameters, refer to the Defining Static and Dynamic Parameters section below. In the example below, the static parameter limit and the dynamic parameters from and triggered are defined. Click New to add a new row for parameters, or click the X to remove a row.
Path Parameters	Enter path parameters supported by the API as Name:Value pairs.
Request Body Parameters	Enter request body parameters supported by the API as Name:Value pairs. The request body is encoded as JSON-formatted plain text. To specify nested JSON entries, use multi-dot notation in the field below.
Response Split Parameter	Enter a split parameter to determine the property within the response body that contains the records. For example, using the split parameter data.results for the Recorded Future log source results in the response below: CODE `{ "data": { "results": [ { "review": { "assignee": null, "statusDate": null, "statusInPortal": "New", "status": "no-action", "noteDate": null, "statusChangeBy": null, "noteAuthor": null, "note": null } } ] } }`

Auth Tab

On the configuration page, click the Auth tab to configure the information below.

Select one of the authentication types below, and then enter the required authentication information.

Authentication Type	Fields
Basic	Select Basic and then enter the following information to use Basic Authentication.
	Username	Enter the basic authentication username.
	Password	Enter the basic authentication password.
XAuth	Select XAuth and then enter the following information to use XAuth Authentication.
	API ID	Enter the ID of the user authenticating via XAuth.
	API Secret Key	Enter the private/secret key associated with the authenticating user or resource.
Token	Select Token and then enter the following information to use Token Authentication.
	Access Token	Enter the value of the access token used to authenticate the log source.
	Token Prefix (Optional.)	Enter the token's prefix value, typically either Bearer or SSWS. The prefix is included in the request before the token value.
	Header Key	Enter the authentication header key to which the prefixed access token will be assigned in the request.
OAuth 2.0	Select OAuth 2.0 and then enter the following information to use OAuth 2.0 Authentication. Axon currently supports only the token base URL form of OAuth 2.0 authentication using Bearer Tokens.
	Token URL	Enter the URL of the authentication server used for validation.
	Auth Prepend Field	Enter the prepend field value used for validation. For example, Bearer.
	Access Token Format	Select either Plain Text or JSON to determine the format of the authentication request.
	Authentication Body Parameters	Optionally specify body parameters as Name: Value pairs to be included in the authentication request.

Pagination Tab

On the configuration page, click the Pagination tab to configure the information below.

Select one of the pagination types below and enter the required information.

Pagination Style	Fields
None	Select to use no pagination style.
Page Link	Select Page Link and then enter the information below to use the page link pagination.
	Cursor Type	Select either URL, Query, or Request Body to determine the next page cursor used. This choice determines which additional options appear.
	Cursor Param Name	Available when Query or Request Body is selected as the Cursor Type. Represents the "key" value for holding the cursor type.
	Cursor Carrier	Choose whether the cursor is located in the Response Body or Response Header.
	Cursor Key Name	Enter the field where the cursor is located within the response body or response header.
	Cursor Key Format Regex	If the cursor is located in the response header, enter the regex pattern for the next response header link. The default value is *<([^<>\\\\s])>.\"next\"*
	Max Number of Pages	Enter the maximum number of pages to be returned in a result.
Record Base	Select Record Base pagination and enter the record parameters as Name: Value pairs.
	Record Parameter	Enter the pagination parameters that come in the API response. For example, Recorded Future's API response has two pagination parameters (return and total). These two parameters are mapped to the pre-defined keys "available_records_key" and "total_records_key".
	Index Start Value	Enter the Index offset value (by default,1) to decide where log collection should begin. This field accepts integer values of 0 or 1.
Page Number	Select Page Number and enter the information below to use page number pagination.
	Cursor Location	Choose whether the cursor is located in the Request Body or Query Parameters.
	Page Number	Enter the page number on which to begin the request.
	Max Number of Pages	Enter the maximum number of pages to be returned in a result.
	Page Number Location	If the cursor is located in the Request Body, enter the complete JSON path for the cursor key name.

Validate the Configuration

Once all four tabs have been configured, click the Send Test Request button to attempt to validate the configuration.

A successful validation displays a green successful GET or POST result (depending on the option selected on the Request tab), a status code of 200, and the length of time, in milliseconds, the response took.

When selecting OAuth 2.0 as the authentication type, two separate tests are shown: one for the token, and one for the actual cloud API response.

Both request headers and response headers are shown in the testing panel:

In the event that the test fails, Axon provides a detailed error message that can help you to resolve the issue(s) with your configuration. The problem field will also be highlighted on the left-hand side of the window. Below are some example errors and the steps needed to resolve them.

Error

Resolution

The specified Response Split Parameter on the Request tab is incorrect.

Correct the response split parameter using the response body provided in the testing panel to determine the property where the records are expected to be found.

The specified Cursor Key Name on the Pagination tab is not available in the API response body.

Correct the cursor key name using the response body provided in the testing panel to determine where the cursor is located.

One or more of the specified Record Parameters on the Pagination tab are not available in the API response body.

Correct the record parameters using the full response body provided in the testing panel to determine the parameters that are available.

Save the Configuration

Once all information has been configured and the configuration has been validated, click Connect to add this generic REST collector to the list of collectors in Axon.

Defining Static and Dynamic Parameters

The parameters used when defining Query Parameters or Request Body Parameters in the Generic REST Collector can be either static or dynamic. This section explains the differences between these parameter types and how to use them in the Generic REST Collector.

Static Parameters

Static parameters are parameters for which the value remains the same throughout all requests. In the example image below, the parameter "limit" is a static parameter because this value will not change across multiple requests.

Dynamic Parameters

Dynamic parameters (also known as template parameters) are parameters for which the value in an API request changes for every collection run on a given frequency and therefore need to be dynamically managed by the system. The following must be taken into account when defining dynamic parameters:

Dynamic parameters are always provided inside curly brackets, like this example: {parameter_name}
In Axon, there are four pre-defined dynamic parameters: {start_time}, {end_time}, {page_offset}, and {record_offset}.
These parameters indicate how to traverse the collection window.
Generally, attributes of an API request related to date, time, offset, etc. are mapped to the pre-defined dynamic parameters listed above.
The user must define dynamic parameters in either Request Body parameters or Query parameters, depending upon the API Request being made.
For defining these parameters, refer to the sample API Request (in some cases, the API Response as well) of the API and identify the mapping of the pre-defined dynamic parameters.

Request Body Dynamic Parameters

The following examples highlight potential dynamic parameters that can be used in the Request Body section of the Generic REST collector.

Actual API Request Body Equivalent Request Body Parameters in the Generic REST Collector Configuration

CODE

"criteria": {
        "last_update_time": {
            "end": "2023-10-13T06:41:00.346Z",
            "start": "2023-10-12T06:41:00.346Z"
        }
    },
    "sort": [
        {
            "field": "create_time",
            "order": "ASC"
        }
    ],
    "start": "1",
    "rows": "50"

In this example, criteria.last_update_time.start and criteria.last_update_time.end are time-related attributes in the API Request. Therefore, these are mapped to the appropriate pre-defined Axon dynamic parameters {start_time} and {end_time}, respectively.

The parameter start indicates the number from which records have to be fetched, meaning it should be increased after each request. Therefore, it has been mapped to the pre-defined Axon dynamic parameter {record_offset}, meaning the value of the start parameter will be increased automatically by the collector.

CODE

{
"filter": "dateTime:['{2023-10-12T06:41:00.346Z}'..'{2023-10-13T06:41:00.346Z}']",
"pageSize": "10",
"pageNumber": "2"
}

In this example, filter contains time-related attributes in the API Request. Therefore, it is mapped to the appropriate pre-defined Axon dynamic parameters {start_time} and {end_time} in the same format.

The parameter pageNumber indicates the number of the page from which records have to be fetched, meaning it should be increased after each request. Therefore, it has been mapped to the pre-defined Axon dynamic parameter {page_offset}, meaning the value of start parameter will be increased automatically by the collector.

Query Parameter Dynamic Parameters

The following examples highlight potential dynamic parameters that can be used in the Query Parameters section of the Generic REST collector.

Example API Request URL	Equivalent Query Parameters in the Generic REST Collector Configuration
https://dev-28433091.okta.com/api/v1/logs?limit=1&since=2023-04-19T15:18:47.495Z	In this example, since contains time-related attributes in the API Request. Therefore, it is mapped to the appropriate Axon pre-defined dynamic parameter {start_time}.
https://tap-api-v2.proofpoint.com/v2/siem/all?format=json&interval=2023-10-19T15:19:47.495Z/2023-10-19T15:20:47.495Z	In this example, interval contains time-related attributes in the API Request. Therefore, it is mapped to the appropriate Axon pre-defined dynamic parameters {start_time} and {end_time} in the same format.
https://api.recordedfuture.com/v3/alerts/?limit=20&from=0&triggered=[2023-01-29T04:42:18.547Z,2023-01-30T04:42:18.547Z]	In this example, triggered contains time-related attributes in the API Request. Therefore, it is mapped to the appropriate Axon pre-defined dynamic parameters {start_time} and {end_time} in the same format. The parameter from indicates the number from which records must be fetched, meaning it should be increased after every request. Therefore, it has been mapped to the Axon pre-defined dynamic parameter {record_offset}, meaning the value of start parameter will be increased automatically by the collector.
https://reports.office365.com/ecp/reportingwebservice/reporting.svc/MessageTrace?$format=json&$filter=StartDate eq datetime'2023-09-25' and EndDate eq datetime'2023-09-26'	In this example, filter contains time-related attributes in the API Request. Therefore, it is mapped to the appropriate Axon pre-defined dynamic parameters {start_time} and {end_time} in the same format.