AWS S3 CloudTrail Events API Collection

AWS CloudTrail provides a management system that enables users to manage and deploy networks at geographically distributed locations. Using the AWS S3 Flat File log source, the System Monitor Agent can collect CloudTrail logs from an S3 bucket that includes numerous logs from multiple regions and accounts. You can also collect logs recursively within a single S3 bucket (logs in subfolders). This document explains how to configure the collection of AWS S3 CloudTrail events using the Web Console's cloud-to-cloud functionality. This feature is available only to LRCloud customers.

Prerequisites

Before configuring collection from AWS, do the following:

• Make sure that the customer is an LRCloud customer and has their environment hosted.
• Ensure that you have a valid AWS Access Key and Secret Access Key.
  
  

The cloud-to-cloud collection uses the AWS ListObjects API to collect logs from AWS S3 CloudTrail sources. The API may not return the full set of logs due to a known limitation in the AWS ListObject API. Requests for logs are returned in a series of transmissions using continuation tokens to keep track of previously collected files. Each continuation token returned by the API is based on the last file collected from the S3 bucket. This functionality can cause logs to be missed if new files added to the S3 bucket are placed before the last file collected from the last continuation token.

For more details on the ListObjects API functionality, see the following links to AWS documentation:

 API List Objects Example 7

API List Objects Example 9

Initialize the Logs Source

1. Log in to the Web Console as a Restricted Administrator User.
2. On the top navigation bar, click the Administration icon 

   

    and select Cloud Log Collection.
3. At the top of the Cloud Log Collection page, click New Log Source.
   The New cloud log collection dialog box appears.
4. Select the AWS S3 CloudTrail SYSMON AGENT tile.
   The Add AWS S3 CloudTrail Events Log Source window appears.
   

   

5. Enter the following details:

   Setting	Description
   Region	The endpoint region code for the specific AWS CloudTrail S3 bucket (for example, us-east-1). For more information, see Amazon S3 Regions and Endpoints.
   Access Key ID	Enter the AWS Access Key ID (for example, AKIAIOSFODNN7EXAMPLE).
   Secret Access Key	Enter the AWS Secret Access Key (for example, wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY).
   Bucket Name	The name of the S3 bucket where logs are stored.
   Log Type	The type of log from which logs are being fetched (for example, CloudTrail or VPCFlowLogs). The log type is case-sensitive. For more information on VPC Flow Logs, see VPC Flow Logs.
   File Path	The absolute path for the log type defined in LogType setting. Example formats: AWSLogs/697238620699/CloudTrail/ AWSLogs/697238620699/VPCFlowLogs/ The file path is case-sensitive.
   Depth To Recurse	The depth of folders where logs are present. Examples: Settings Examples of File Path DepthToRecurse=1 FilePath=AWSLogs AWSLogs/697238620698/ AWSLogs/697238620699/ DepthToRecurse=2 FilePath=AWSLogs AWSLogs/697238620698/CloudTrail/ AWSLogs/697238620699/CloudTrail/ DepthToRecurse=3 FilePath=AWSLogs AWSLogs/697238620698/CloudTrail/Region-1 AWSLogs/697238620698/CloudTrail/Region-2 AWSLogs/697238620698/CloudTrail/Region-3 AWSLogs/697238620699/CloudTrail/Region-1 AWSLogs/697238620699/CloudTrail/Region-2 AWSLogs/697238620699/CloudTrail/Region-3
   Exclusion Directories	One or more directories that you want to exclude from collection. If you want to exclude multiple directories, separate them with a comma. If you do not want to use this setting, leave it blank. Example scenario: Your AWS S3 bucket contains three directories, but you want to collect from only one of them. Directories in Bucket Setting Directories Excluded AWSLogs/697238620698/CloudTrail-Digest/ AWSLogs/697238620698/CloudTrail-Insight/ AWSLogs/697238620698/CloudTrail/ ExclusionDirectories=CloudTrail-Digest,CloudTrail-Insight AWSLogs/697238620698/CloudTrail-Digest/ AWSLogs/697238620698/CloudTrail-Insight/ If you set ExclusionDirectories=CloudTrail, you will exclude all directories containing CloudTrail in their name. The directory name is case-sensitive.
   Inclusions	One or more file extensions that you want to collect (for example, *.gz or *.txt). If you want to include multiple file extensions, separate them with a comma (for example, *.gz,*.txt). You should not change the value for this field.
   Exclusions	One or more file extensions that you want to exclude from collection (for example, *.gz or *.txt). If you want to exclude multiple file extensions, separate them with a comma (for example, *.gz,*.txt). You should not change the value for this field.

6. Click Save.

A new active log source is created and accepted in the Client Console with the provided information. Collection should start automatically in few minutes.

The Platform Manager hosts all the log sources. It is recommended to create a new host entity and move the log source to the new host.

For security, the values entered are encrypted using LRCrypt.

Default Config Values for AWS S3 Server Access Events Log Source