Skip to main content
Skip table of contents

(LRCloud Only) Configure AWS S3 CloudTrail Events Using Cloud to Cloud

AWS CloudTrail provides a management system that enables users to manage and deploy networks at geographically distributed locations. Using the AWS S3 Flat File log source, the System Monitor Agent can collect CloudTrail logs from an S3 bucket that includes numerous logs from multiple regions and accounts. You can also collect logs recursively within a single S3 bucket (logs in subfolders). This document explains how to configure the collection of AWS S3 CloudTrail events using the web console's cloud to cloud functionality. This is available to LRCloud customers only.

Prerequisites

Before you start to configure collection from AWS, you must ensure you have the following:

  • Customer is an LRCloud customer and has their environment hosted
  • A valid AWS Access Key and Secret Access Key

The Cloud to Cloud collection  uses the AWS ListObjects API to collect logs from AWS S3 CloudTrail sources. The API may not return the full set of logs due to a known limitation in the AWS ListObject API. Requests for logs are returned in a series of transmissions using continuation tokens to keep track of previously collected files. Each continuation token returned by the API is based on the last file collected from the S3 bucket. This functionality can cause logs to be missed if new files added to the S3 bucket are placed before the last file collected from the last continuation token (see image below).

For more details on the ListObjects API functionality, see the following links to AWS documentation:

https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjects.html#API_ListObjects_Example_7  

https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjectsV2.html#API_ListObjectsV2_Example_9

Initialize the Logs Source

  1. Log into the web console as an Restricted Administrator User.
  2. On the top navigation bar, click the Administration icon, and select Cloud Log Collection.
  3. At the top of the page, click New Log Source.
  4. Select the tile for AWS S3 CloudTrail Sysmon Agent
  5. Enter the following details:

    SettingDescription
    Region

    The endpoint region code for the specific AWS CloudTrail S3 bucket (for example, us-east-1). For more information, refer to Amazon S3 Regions and Endpoints.

    Access Key IdEnter the AWS Access Key ID. for example, AKIAIOSFODNN7EXAMPLE
    Secret Access KeyEnter the AWS Secret Access Key for example, wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
    Bucket NameThe name of the S3 bucket where logs are stored.
    Log Type

    The type of log from which logs are being fetched. For example, CloudTrail or VPCFlowLogs.

    The log type is case sensitive.

    For more information on VPC Flow Logs, see VPC Flow Logs.

    File Path
    The absolute path for the log type defined in LogType setting. Example formats:

    AWSLogs/697238620699/CloudTrail/

    AWSLogs/697238620699/VPCFlowLogs/

    The file path is case sensitive.

    Depth To Recurse

    The depth of folders where logs are actually present.

    Examples:

    SettingsExamples of File Path

    DepthToRecurse=1

    FilePath=AWSLogs

    AWSLogs/697238620698/

    AWSLogs/697238620699/

    DepthToRecurse=2

    FilePath=AWSLogs

    AWSLogs/697238620698/CloudTrail/

    AWSLogs/697238620699/CloudTrail/

    DepthToRecurse=3

    FilePath=AWSLogs

    AWSLogs/697238620698/CloudTrail/Region-1

    AWSLogs/697238620698/CloudTrail/Region-2

    AWSLogs/697238620698/CloudTrail/Region-3

    AWSLogs/697238620699/CloudTrail/Region-1

    AWSLogs/697238620699/CloudTrail/Region-2

    AWSLogs/697238620699/CloudTrail/Region-3

    Exclusion Directories

    One or more directories that you want to exclude from collection. If you want to exclude multiple directories, separate them with a comma. If you do not want to use this setting, leave it blank.

    Example scenario: Your AWS S3 bucket contains three directories, but you want to collect from only one of them.

    Directories in BucketSettingDirectories Excluded

    AWSLogs/697238620698/CloudTrail-Digest/

    AWSLogs/697238620698/CloudTrail-Insight/

    AWSLogs/697238620698/CloudTrail/

    ExclusionDirectories=CloudTrail-Digest,CloudTrail-Insight

    AWSLogs/697238620698/CloudTrail-Digest/

    AWSLogs/697238620698/CloudTrail-Insight/


    If you set ExclusionDirectories=CloudTrail, you will exclude all directories containing CloudTrail in their name.


    The directory name is case sensitive.

    Inclusions

    One or more file extensions that you want to collect (for example, *.gz or *.txt). If you want to include multiple file extensions, separate them with a comma (for example, *.gz,*.txt).

    You should not change the value for this field.

    Exclusions

    One or more file extensions that you want to exclude from collection (for example, *.gz or *.txt). If you want to exclude multiple file extensions, separate them with a comma (for example, *.gz,*.txt).

    You should not change the value for this field.

  6. Press the Save button

Using the information provided a new active log source is created and accepted in the client console. Collection should start automatically within a couple of minutes.

The log source's host is the Platform Manager; however it is recommended that a new host entity is created and the log sources is moved to the new host.


For security purposes the values entered are stored in encrypted format using LRCrypt.

Default Config Values for AWS S3 Server Access Events Log Source

SettingDefault Value
NoOfBackDaysData1
LogApiRequestsfalse
DataFolderFilesCount100
MaxQueueCount50000

MaxResultCount

100

StartupDelayInSeconds

30
JavaScript errors detected

Please note, these errors can depend on your browser setup.

If this problem persists, please contact our support.