API - AWS S3 CloudTrail (via Flat File)

AWS CloudTrail provides a management system that enables users to manage and deploy networks at geographically distributed locations. Using the AWS S3 Flat File log source, the System Monitor Agent can collect CloudTrail logs from an S3 bucket that includes numerous logs from multiple regions and accounts. You can also collect logs recursively within a single S3 bucket (logs in subfolders). This section explains how to configure the collection of AWS S3 CloudTrail events via the System Monitor.

The LogRhythm System Monitor Agent uses the AWS ListObjects API to collect logs from AWS S3 CloudTrail sources. The API may not return the full set of logs due to a known limitation in the AWS ListObject API. Requests for logs are returned in a series of transmissions using continuation tokens to keep track of previously collected files. Each continuation token returned by the API is based on the last file collected from the S3 bucket. This functionality can cause logs to be missed if new files added to the S3 bucket are placed before the last file collected from the last continuation token (see image below).

For more details on the ListObjects API functionality, see the following links to AWS documentation:

https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjects.html#API_ListObjects_Example_7

https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjectsV2.html#API_ListObjectsV2_Example_9

Configure the awsS3CloudTrail.ini File

A LogRhythm System Monitor Agent is required to collect log files. It needs a user account with access to the AWS API. With the credentials of the AWS IAM user created in the previous section, the awsS3CloudTrail.ini file is used to create a secure connection between the System Monitor and AWS S3.

The awsS3CloudTrail.ini file contains many settings. The table below lists the settings with the default value, the range of values when applicable, and a brief description of the value.

Setting

Range

Default Value

Description

Region

CHANGE_THIS

The endpoint region code for the specific AWS CloudTrail S3 bucket (for example, us-east-1). For more information, refer to Amazon S3 Regions and Endpoints.

AccessKeyId

CHANGE_THIS

The AWS Access Key ID (see note below).

SecretAccessKey

CHANGE_THIS

The AWS Secret Access Key (see note below).

The Access Key ID and Secret Access Key must be encrypted using the lrcrypt command line utility, located in the System Monitor installation directory. See the LogRhythm Password Encryption section of Deployment Security for more information. You must manually paste the encrypted values into the configuration file.

BucketName

CHANGE_THIS

The name of the S3 bucket where logs are stored.

LogType

CHANGE_THIS

The type of log from which logs are being fetched. For example, CloudTrail or VPCFlowLogs.

The log type is case sensitive.

For more information on VPC Flow Logs, see section below.

FilePath

CHANGE_THIS

The absolute path for the log type defined in LogType setting. Example formats:

AWSLogs/697238620699/CloudTrail/

AWSLogs/697238620699/VPCFlowLogs/

The file path is case sensitive.

DepthToRecurse

1-10

The depth of folders where logs are actually present.

Examples:

Settings

Examples of File Path

DepthToRecurse=1

FilePath=AWSLogs

AWSLogs/697238620698/

AWSLogs/697238620699/

DepthToRecurse=2

FilePath=AWSLogs

AWSLogs/697238620698/CloudTrail/

AWSLogs/697238620699/CloudTrail/

DepthToRecurse=3

FilePath=AWSLogs

AWSLogs/697238620698/CloudTrail/Region-1

AWSLogs/697238620698/CloudTrail/Region-2

AWSLogs/697238620698/CloudTrail/Region-3

AWSLogs/697238620699/CloudTrail/Region-1

AWSLogs/697238620699/CloudTrail/Region-2

AWSLogs/697238620699/CloudTrail/Region-3

As of LogRhythm SIEM 7.20, with System Monitor Agents upgrading to .NET 8, this field is no longer included/supported in .ini files.

NoOfBackDaysData

1-7

Number of days back you want to fetch data.

ExclusionDirectories

Blank

One or more directories that you want to exclude from collection. If you want to exclude multiple directories, separate them with a comma. If you do not want to use this setting, leave it blank.

Example scenario: Your AWS S3 bucket contains three directories, but you want to collect from only one of them.

Directories in Bucket

Setting

Directories Excluded

AWSLogs/697238620698/CloudTrail-Digest/

AWSLogs/697238620698/CloudTrail-Insight/

AWSLogs/697238620698/CloudTrail/

ExclusionDirectories=CloudTrail-Digest,CloudTrail-Insight

AWSLogs/697238620698/CloudTrail-Digest/

AWSLogs/697238620698/CloudTrail-Insight/

If you set ExclusionDirectories=CloudTrail, you will exclude all directories containing CloudTrail in their name.

The directory name is case sensitive.

As of LogRhythm SIEM 7.20, with System Monitor Agents upgrading to .NET 8, this field is no longer included/supported in .ini files.

DataFolderFilesCount

100-1000

100

Throttling data folder size. The Agent stops downloading new compressed files from AWS once the file count reaches the specified value.

This setting should only be used by customers experiencing a very high backlog of logs on the Agent.

MaxQueueCount

50000-100000

50000

Throttling log queue size.

You should not change the value for this field.

Inclusions

*.gz

One or more file extensions that you want to collect (for example, *.gz or *.txt). If you want to include multiple file extensions, separate them with a comma (for example, *.gz,*.txt).

You should not change the value for this field.

Exclusions

CHANGE_THIS

One or more file extensions that you want to exclude from collection (for example, *.gz or *.txt). If you want to exclude multiple file extensions, separate them with a comma (for example, *.gz,*.txt).

You should not change the value for this field.

MaxResultCount

1–100

100

The number of objects to fetch in a single request.

StartupDelayInSeconds

0-300

If the API needs to be queried when the System Monitor is started, it will wait this long before running.

LogApiRequests

false

Enables (true) or disables (false) diagnostic logging of HTTP and HTTPS requests to the API.

API request logging should only be used with assistance from LogRhythm Customer Support. You should leave this field unchanged (false).

RegionFolders

Blank

Add all regions from which logs will be collected, separated by commas. These regions can be located in the CloudTrail objects folder:

For example, to configure this field for only the regions shown in the screenshot above:

# Case-Sensitive and names should contain whole world
# See common regions names below
RegionFolders=ap-northeast-1,ap-northeast-2,ap-northeast-3,ap-south-1,ap-southeast-1,ap-southeast-2,ca-central-1,eu-central-1,eu-north-1,eu-west-1,eu-west-2,eu-west-3

These regions are case-sensitive.

Proxy Settings

ProxyServer

The IP address or DNS name of a proxy server to use for connecting to AWS.

ProxyPort

The port to use on the proxy server.

UserName

The user name to send if authentication is required on the proxy server.

Password

The password for the specified user name.

If authentication is required, the password must be encrypted using the lrcrypt command line utility, located in the System Monitor installation directory. See LogRhythm Password Encryption for more information. You must manually paste the encrypted values into the configuration file.

Domain

The domain to use for connecting to the proxy server.

(Optional) Settings – You should not change any of the values in this section

SignatureVersion

Indicates the authentication scheme in use – do not change.

Edit the awsS3CloudTrail.ini file with the appropriate credentials and information to create a secure connection between the System Monitor and AWS S3.

Before you begin these instructions, ensure that you have the Access Key ID and the Secret Access Key. These keys are needed to configure the awsS3CloudTrail.ini file.

Open Windows Explorer and go to the following directory: C:\Program Files\LogRhythm\LogRhythm System Monitor\config
Open awsS3CloudTrail.ini with a text editor.
Most of the configuration can be used as is. A few of the settings need to be changed so the LogRhythm Agent can access the CloudTrail instance to collect log files.
For Region, replace CHANGE_THIS with the "Region" ID for the AWS CloudTrail S3 bucket — for example, us-east-1. For more information, refer to Amazon S3 Regions and Endpoints.
For AccessKeyId, replace CHANGE_THIS with the Access Key generated when you created the IAM user for this instance of CloudTrail — encrypt with lrcrypt before adding to the INI file.
For SecretAccessKey, replace CHANGE_THIS with the Secret Access Key generated when you created the IAM user for this instance of CloudTrail — encrypt with lrcrypt before adding to the INI file.

The AccessKeyId and SecretAccessKey values must be encrypted using the lrcrypt command line utility.
For BucketName, replace CHANGE_THIS with the name of the S3 bucket where logs are stored.
If you do not want to scan all directories within the specified S3 bucket, you can use FilePath to specify the top-level directory that contains CloudTrail logs that you want to collect.
Save and close the file.

After you configure the device, you must also configure LogRhythm according to the instructions provided on the overview page of this guide. Only Global Admins or Restricted Admins with elevated View and Manage privileges can take this action.

The name of the log message source is API - AWS S3 Flat File. In addition, when configuring this log source:

For Log Message Processing Mode, select MPE Processing Enabled, Event Forwarding Enabled.
For Log Message Processing Engine (MPE) Policy, select LogRhythm Default.

VPC Flow Logs

VPC Flow Logs is a feature that allows you to monitor IP traffic associated with your Amazon VPC (Virtual Private Cloud).

For additional information on VPC Flow Logs, see AWS documentation.

VPC Flow Logs: Currently Supported Log Types

Type	Product Version	Supported Schema Fields
Traffic	N/A	<account-id>, <action>, <bytes>, <dstaddr>, <dstport>, <end>, <interface-id>, <log-status>, <packets>, <protocol>, <srcaddr>, <srcport>, <start>, <version>

VPC Flow Logs: Parsed Metadata Fields

Device Field Name	LogRhythm Metadata Field	Value/Data Type
action	<action>, <tag1>	Text/String
account-id	<account>	Text/String
bytes	<bytes>	Numeric
dstaddr	<dip>	IP Address
dstport	<dport>	Numeric
interface-id	<dinterface>	Numeric/Text/String
log-status	<status>, <tag2>	Text/String
packets	<packets>	Numeric
protocol	<protnum>	Numeric
srcaddr	<sip>	IP Address
srcport	<sport>	Numeric
version	<version>	Text/String