API - AWS S3 CloudTrail (via Flat File)

AWS CloudTrail provides a management system that enables users to manage and deploy networks at geographically distributed locations. Using the AWS S3 Flat File log source, the System Monitor Agent can collect CloudTrail logs from an S3 bucket that includes numerous logs from multiple regions and accounts. You can also collect logs recursively within a single S3 bucket (logs in subfolders). This section explains how to configure the collection of AWS S3 CloudTrail events via the System Monitor.

The LogRhythm System Monitor Agent uses the AWS ListObjects API to collect logs from AWS S3 CloudTrail sources. The API may not return the full set of logs due to a known limitation in the AWS ListObject API. Requests for logs are returned in a series of transmissions using continuation tokens to keep track of previously collected files. Each continuation token returned by the API is based on the last file collected from the S3 bucket. This functionality can cause logs to be missed if new files added to the S3 bucket are placed before the last file collected from the last continuation token (see image below).

For more details on the ListObjects API functionality, see the following links to AWS documentation:

https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjects.html#API_ListObjects_Example_7  

https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjectsV2.html#API_ListObjectsV2_Example_9

Configure the awsS3CloudTrail.ini File

A LogRhythm System Monitor Agent is required to collect log files. It needs a user account with access to the AWS API. With the credentials of the AWS IAM user created in the previous section, the awsS3CloudTrail.ini file is used to create a secure connection between the System Monitor and AWS S3.

The awsS3CloudTrail.ini file contains many settings. The table below lists the settings with the default value, the range of values when applicable, and a brief description of the value.

Setting	Range	Default Value	Description
Region		CHANGE_THIS	The endpoint region code for the specific AWS CloudTrail S3 bucket (for example, us-east-1). For more information, refer to Amazon S3 Regions and Endpoints.
AccessKeyId		CHANGE_THIS	The AWS Access Key ID (see note below).
SecretAccessKey		CHANGE_THIS	The AWS Secret Access Key (see note below).
The Access Key ID and Secret Access Key must be encrypted using the lrcrypt command line utility, located in the System Monitor installation directory. See LogRhythm Password Encryption for more information. You must manually paste the encrypted values into the configuration file.
BucketName		CHANGE_THIS	The name of the S3 bucket where logs are stored.
LogType		CHANGE_THIS	The type of log from which logs are being fetched. For example, CloudTrail or VPCFlowLogs. The log type is case sensitive. For more information on VPC Flow Logs, see section below.
FilePath		CHANGE_THIS	The absolute path for the log type defined in LogType setting. Example formats: AWSLogs/697238620699/CloudTrail/ AWSLogs/697238620699/VPCFlowLogs/ The file path is case sensitive.
DepthToRecurse	1-10	1	The depth of folders where logs are actually present. Examples: Settings Examples of File Path DepthToRecurse=1 FilePath=AWSLogs AWSLogs/697238620698/ AWSLogs/697238620699/ DepthToRecurse=2 FilePath=AWSLogs AWSLogs/697238620698/CloudTrail/ AWSLogs/697238620699/CloudTrail/ DepthToRecurse=3 FilePath=AWSLogs AWSLogs/697238620698/CloudTrail/Region-1 AWSLogs/697238620698/CloudTrail/Region-2 AWSLogs/697238620698/CloudTrail/Region-3 AWSLogs/697238620699/CloudTrail/Region-1 AWSLogs/697238620699/CloudTrail/Region-2 AWSLogs/697238620699/CloudTrail/Region-3
NoOfBackDaysData	1-7	1	Number of days back you want to fetch data.
ExclusionDirectories		Blank	One or more directories that you want to exclude from collection. If you want to exclude multiple directories, separate them with a comma. If you do not want to use this setting, leave it blank. Example scenario: Your AWS S3 bucket contains three directories, but you want to collect from only one of them. Directories in Bucket Setting Directories Excluded AWSLogs/697238620698/CloudTrail-Digest/ AWSLogs/697238620698/CloudTrail-Insight/ AWSLogs/697238620698/CloudTrail/ ExclusionDirectories=CloudTrail-Digest,CloudTrail-Insight AWSLogs/697238620698/CloudTrail-Digest/ AWSLogs/697238620698/CloudTrail-Insight/ If you set ExclusionDirectories=CloudTrail, you will exclude all directories containing CloudTrail in their name. The directory name is case sensitive.
DataFolderFilesCount	100-1000	100	Throttling data folder size. The Agent stops downloading new compressed files from AWS once the file count reaches the specified value. This setting should only be used by customers experiencing a very high backlog of logs on the Agent.
MaxQueueCount	50000-100000	50000	Throttling log queue size. You should not change the value for this field.
Inclusions		*.gz	One or more file extensions that you want to collect (for example, *.gz or *.txt). If you want to include multiple file extensions, separate them with a comma (for example, *.gz,*.txt). You should not change the value for this field.
Exclusions		CHANGE_THIS	One or more file extensions that you want to exclude from collection (for example, *.gz or *.txt). If you want to exclude multiple file extensions, separate them with a comma (for example, *.gz,*.txt). You should not change the value for this field.
MaxResultCount	1–100	100	The number of objects to fetch in a single request.
StartupDelayInSeconds	0-300	30	If the API needs to be queried when the System Monitor is started, it will wait this long before running.
LogApiRequests		false	Enables (true) or disables (false) diagnostic logging of HTTP and HTTPS requests to the API. API request logging should only be used with assistance from LogRhythm Customer Support. You should leave this field unchanged (false).
Proxy Settings
ProxyServer	The IP address or DNS name of a proxy server to use for connecting to AWS.
ProxyPort	The port to use on the proxy server.
UserName	The user name to send if authentication is required on the proxy server.
Password	The password for the specified user name. If authentication is required, the password must be encrypted using the lrcrypt command line utility, located in the System Monitor installation directory. See LogRhythm Password Encryption for more information. You must manually paste the encrypted values into the configuration file.
Domain	The domain to use for connecting to the proxy server.
(Optional) Settings – You should not change any of the values in this section
SignatureVersion	Indicates the authentication scheme in use – do not change.

Edit the awsS3CloudTrail.ini file with the appropriate credentials and information to create a secure connection between the System Monitor and AWS S3.

Before you begin these instructions, ensure that you have the Access Key ID and the Secret Access Key. These keys are needed to configure the awsS3CloudTrail.ini file.

1. Open Windows Explorer and go to the following directory: C:\Program Files\LogRhythm\LogRhythm System Monitor\config
2. Open awsS3CloudTrail.ini with a text editor.
   Most of the configuration can be used as is. A few of the settings need to be changed so the LogRhythm Agent can access the CloudTrail instance to collect log files.

3. For Region, replace CHANGE_THIS with the "Region" ID for the AWS CloudTrail S3 bucket — for example, us-east-1. For more information, refer to Amazon S3 Regions and Endpoints.

4. For AccessKeyId, replace CHANGE_THIS with the Access Key generated when you created the IAM user for this instance of CloudTrail — encrypt with lrcrypt before adding to the INI file.

5. For SecretAccessKey, replace CHANGE_THIS with the Secret Access Key generated when you created the IAM user for this instance of CloudTrail — encrypt with lrcrypt before adding to the INI file.

   

   The AccessKeyId and SecretAccessKey values must be encrypted using the lrcrypt command line utility.

6. For BucketName, replace CHANGE_THIS with the name of the S3 bucket where logs are stored.
7. If you do not want to scan all directories within the specified S3 bucket, you can use FilePath to specify the top-level directory that contains CloudTrail logs that you want to collect.
8. Save and close the file.

After you configure the device, you must also configure LogRhythm according to the instructions provided on the overview page of this guide. Only Global Admins or Restricted Admins with elevated View and Manage privileges can take this action.

The name of the log message source is API - AWS S3 Flat File. In addition, when configuring this log source:

• For Log Message Processing Mode, select MPE Processing Enabled, Event Forwarding Enabled.
• For Log Message Processing Engine (MPE) Policy, select LogRhythm Default.

VPC Flow Logs

VPC Flow Logs is a feature that allows you to monitor IP traffic associated with your Amazon VPC (Virtual Private Cloud).

For additional information on VPC Flow Logs, see AWS documentation.

VPC Flow Logs: Currently Supported Log Types

VPC Flow Logs: Parsed Metadata Fields