(LRCloud Only) Configure AWS S3 CloudTrail Events Using Cloud to Cloud

AWS CloudTrail provides a management system that enables users to manage and deploy networks at geographically distributed locations. Using the AWS S3 Flat File log source, the System Monitor Agent can collect CloudTrail logs from an S3 bucket that includes numerous logs from multiple regions and accounts. You can also collect logs recursively within a single S3 bucket (logs in subfolders). This document explains how to configure the collection of AWS S3 CloudTrail events using the web console's cloud to cloud functionality. This is available to LRCloud customers only.

Prerequisites

Before you start to configure collection from AWS, you must ensure you have the following:

• Customer is an LRCloud customer and has their environment hosted
• A valid AWS Access Key and Secret Access Key

The Cloud to Cloud collection  uses the AWS ListObjects API to collect logs from AWS S3 CloudTrail sources. The API may not return the full set of logs due to a known limitation in the AWS ListObject API. Requests for logs are returned in a series of transmissions using continuation tokens to keep track of previously collected files. Each continuation token returned by the API is based on the last file collected from the S3 bucket. This functionality can cause logs to be missed if new files added to the S3 bucket are placed before the last file collected from the last continuation token (see image below).

For more details on the ListObjects API functionality, see the following links to AWS documentation:

https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjects.html#API_ListObjects_Example_7  

https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjectsV2.html#API_ListObjectsV2_Example_9

Initialize the Logs Source

1. Log into the web console as an Restricted Administrator User.
2. On the top navigation bar, click the Administration icon, and select Cloud Log Collection.
3. At the top of the page, click New Log Source.
4. Select the tile for AWS S3 CloudTrail Sysmon Agent
   

   

5. Enter the following details:

   Setting	Description
   Region	The endpoint region code for the specific AWS CloudTrail S3 bucket (for example, us-east-1). For more information, refer to Amazon S3 Regions and Endpoints.
   Access Key Id	Enter the AWS Access Key ID. for example, AKIAIOSFODNN7EXAMPLE
   Secret Access Key	Enter the AWS Secret Access Key for example, wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
   Bucket Name	The name of the S3 bucket where logs are stored.
   Log Type	The type of log from which logs are being fetched. For example, CloudTrail or VPCFlowLogs. The log type is case sensitive. For more information on VPC Flow Logs, see VPC Flow Logs.
   File Path	The absolute path for the log type defined in LogType setting. Example formats: AWSLogs/697238620699/CloudTrail/ AWSLogs/697238620699/VPCFlowLogs/ The file path is case sensitive.
   Depth To Recurse	The depth of folders where logs are actually present. Examples: Settings Examples of File Path DepthToRecurse=1 FilePath=AWSLogs AWSLogs/697238620698/ AWSLogs/697238620699/ DepthToRecurse=2 FilePath=AWSLogs AWSLogs/697238620698/CloudTrail/ AWSLogs/697238620699/CloudTrail/ DepthToRecurse=3 FilePath=AWSLogs AWSLogs/697238620698/CloudTrail/Region-1 AWSLogs/697238620698/CloudTrail/Region-2 AWSLogs/697238620698/CloudTrail/Region-3 AWSLogs/697238620699/CloudTrail/Region-1 AWSLogs/697238620699/CloudTrail/Region-2 AWSLogs/697238620699/CloudTrail/Region-3
   Exclusion Directories	One or more directories that you want to exclude from collection. If you want to exclude multiple directories, separate them with a comma. If you do not want to use this setting, leave it blank. Example scenario: Your AWS S3 bucket contains three directories, but you want to collect from only one of them. Directories in Bucket Setting Directories Excluded AWSLogs/697238620698/CloudTrail-Digest/ AWSLogs/697238620698/CloudTrail-Insight/ AWSLogs/697238620698/CloudTrail/ ExclusionDirectories=CloudTrail-Digest,CloudTrail-Insight AWSLogs/697238620698/CloudTrail-Digest/ AWSLogs/697238620698/CloudTrail-Insight/ If you set ExclusionDirectories=CloudTrail, you will exclude all directories containing CloudTrail in their name. The directory name is case sensitive.
   Inclusions	One or more file extensions that you want to collect (for example, *.gz or *.txt). If you want to include multiple file extensions, separate them with a comma (for example, *.gz,*.txt). You should not change the value for this field.
   Exclusions	One or more file extensions that you want to exclude from collection (for example, *.gz or *.txt). If you want to exclude multiple file extensions, separate them with a comma (for example, *.gz,*.txt). You should not change the value for this field.

6. Press the Save button

Using the information provided a new active log source is created and accepted in the client console. Collection should start automatically within a couple of minutes.

The log source's host is the Platform Manager; however it is recommended that a new host entity is created and the log sources is moved to the new host.

For security purposes the values entered are stored in encrypted format using LRCrypt.

Default Config Values for AWS S3 Server Access Events Log Source