Skip to main content
Skip table of contents

Configure Amazon AWS

Overview

  • The Amazon S3 Collector supports log collection from multiple S3 buckets and AWS accounts.

  • Each AWS S3 bucket from which you want to collect logs should be configured to send Object Create Events to a Simple Queue Service (SQS) queue. Then, you provide the queue name(s) and region(s) to the Amazon S3 Collector.

  • The Amazon S3 Collector offers one authentication method: key-based security credentials (access key and secret key).

  • The Amazon S3 Collector supports the use of the Dead Letter Queue service in SQS.  This allows problematic messages to be removed from the SQS queue and stored for later analysis.

Log Source Types

The Amazon S3 Collector collects logs for any file format, but it supports decompression of the .gz file format. Only single-level depth decompression is supported for any file format.

S3 buckets can contain a variety of log source types.  Currently, files in S3 buckets are assumed to contain many single-line logs, delimited by a newline character (\n).

Access and Configure AWS S3 and SQS

This section provides information for accessing and configuring your AWS S3 instance.

Create a Simple Queue Service (SQS) in AWS (Role-Based Authentication)

  1. Login to the AWS account at:

    CODE 
    https://console.aws.amazon.com/

  2. On the top menu bar, click Services.

  3. In the search field, enter sqs, and then select Simple Queue Service from the suggested search results.

    image-20240301-235132.png

  4. In the upper-right corner, click Create queue.

    image-20240301-235151.png

  5. Select Standard from the options.

    image-20240301-235204.png

  6. Enter a name for the queue, and then copy and paste it into a text document for use in future steps.

  7. Keep the default settings in the Configuration and Encryption sections.

    image-20240306-200831.png

  8. When prompted to Choose method, select Advanced.

    image-20240301-235218.png

  9. Edit the following lines of the policy document.
    Alternatively, to use the policy generator feature to create a policy, skip to step 10.

    1. Replace the value after “Resource”: with your SQS ARN. 

    2. Replace the value after “aws:SourceArn”: with your bucket name. 

    3. Replace the value after “aws:SourceAccount”: with the account ID where the bucket resides.

      TEXT 
      {
  "Version": "2012-10-17",
  "Id": "Policy1709033562483",
  "Statement": [
    {
      "Sid": "Stmt1709033540226",
      "Effect": "Allow",
      "Principal": {
        "Service": "s3.amazonaws.com"
      },
      "Action": [
        "sqs:DeleteMessage",
        "sqs:GetQueueUrl",
        "sqs:ReceiveMessage",
        "sqs:SendMessage",
        "sqs:GetQueueAttributes"
      ],
      "Resource": "{SQS-queue-ARN value}",
      "Condition": {
        "StringEquals": {
          "aws:SourceAccount": "{bucket-owner-account-id}"
        },
        "ArnLike": {
          "aws:SourceArn": "arn:aws:s3:::{bucket-name value}"
        }
      }
    }
  ]
}

  10. (Optional - if you created the policy manually in step 9, skip ahead to step 22.) Click Policy generator.
    The AWS Policy Generator page opens.

    image-20240301-235231.png

  11. From the Select Type of Policy drop-down menu, select SQS Queue Policy.

    image-20240301-235244.png

  12. For the Effect option, choose Allow.

  13. In the Principal field, specify any of the following principals for the policy:

    1. AWS account and root user

    2. IAM roles

    3. Role sessions

    4. IAM users

    5. Federated user sessions

    6. AWS services

    7. All principals

Most commonly, the Principal will be the Account ID or s3.amazonaws.com.

See Specifying a Principal in the AWS documentation for more information on choosing a principal.

  1. From the Actions drop-down menu, select Send Message, DeleteMessage, GetQueueURL, GetQueueAttributes, and ReceiveMessage.

    image-20240301-235029.png

  2. In the Amazon Resource Name (ARN) field, enter your ARN in the following format, replacing <region> with your S3 bucket region, <account_ID> with your account ID, and <queue_name> with your queue name.

CODE 
arn:aws:sqs:<region>:<account_ID>:<queue_name>

  1. Click Add conditions.

    1. From the Condition drop-down menu, select ArnLike.

    2. From the Key drop-down menu, select aws:SourceArn.

    3. In the Value field, enter arn:aws:s3:::awsexamplebucket1, replacing awsexamplebucket1 with your bucket name.

    4. Click Add Condition.

      image-20240301-235303.png

  2. Click Add conditions again.

    1. From the Condition drop-down menu, select StringEquals.

    2. From the Key drop-down menu, select aws:sourceAccount.

    3. In the Value field, enter your account ID.

    4. Click Add Condition.

      image-20240301-235315.png

  3. Click Add Statement.

  4. Click Generate Policy.

    image-20240301-235328.png

The Generated policy should look like the following: 

TEXT 
{
  "Version": "2012-10-17",
  "Id": "Policy1709033562483",
  "Statement": [
    {
      "Sid": "Stmt1709033540226",
      "Effect": "Allow",
      "Principal": {
        "Service": "s3.amazonaws.com"
      },
      "Action": [
        "sqs:DeleteMessage",
        "sqs:GetQueueUrl",
        "sqs:ReceiveMessage",
        "sqs:SendMessage",
        "sqs:GetQueueAttributes"
      ],
      "Resource": "arn:aws:sqs:us-east-1:0123456789:IntegrationEngTestQueue",
      "Condition": {
        "StringEquals": {
          "aws:SourceAccount": "0123456789"
        },
        "ArnLike": {
          "aws:SourceArn": "arn:aws:s3:::integrationaxonbucket1"
        }
      }
    }
  ]
}

Replace the key "AWS" with "Service" in the Principal line of the generated policy.

The Account ID and other details shown in the above example policy are sample values.

  1. Copy the generated policy. 

  2. Go back to the SQS policy tab, and paste the policy there.

  3. Click Create queue.

    image-20240301-235349.png

    The SQS queue is added to the list of queues, and the SQS Queue URL required to Configure the Amazon S3 Collector in Axon can be obtained.

Configure Events to be Sent to SQS Queues

  1. Return to the AWS account homepage.

  2. On the menu bar at the top, click Services.

  3. In the search bar, enter s3, and then select S3 (Scalable Storage in the Cloud) from the suggested search results.

    image-20240301-235402.png

  4. Search for the bucket from which you want to get events.

    image-20240301-235413.png

  5. Click the name of the bucket, and then click the Properties tab.

    image-20240301-235423.png

  6. Under Event notifications, click Create event notification.

    image-20240301-235434.png

  7. Configure the following permissions:

    1. Name. Enter any name.

    2. Events. Select the events about which you need to be notified. For example, All object create events.

    3. Send to. Enter the destination queue (created in the previous procedure) where you want to send events.

    4. SQS. Select the queue created in the previous procedure.

  8. Click Save changes.

    image-20240301-235449.png


    If the event has been configured correctly, the tile displays active notifications.

    image-20240301-235500.png

  9. Click the Permissions tab to set bucket ownership of objects.

    image-20240301-235511.png

  10. Scroll down and under the Object ownership section, click Edit.

    image-20240301-235523.png

  11. Select ACLs enabled, acknowledge that ACLs will be restored, and then click Bucket owner preferred.

    image-20240301-235535.png

  12. Click Save changes.                  

Create and Configure the SQS Dead Letter Queue

Prerequisites

  • You must have an existing SQS queue, configured via the steps described above in Access and Configure AWS S3 and SQS.

  • The dead letter queue and the SQS queue sending logs to LogRhythm must be in the same region.

  • The dead letter queue must be configured as a "standard" queue.

Create a Dead Letter Queue and Associate with an Existing SQS Queue

  1. Login to the AWS account at:

    CODE 
    https://console.aws.amazon.com/

  2. On the top menu bar, click Services.

  3. In the search field, enter sqs, and then select Simple Queue Service from the suggested search results.

    image-20240301-235547.png

  4. In the navigation pane, choose Queues.

  5. Click the Create queue button. 

    image-20240301-235600.png

  6. Create a new Standard queue, with default settings.         

  7. Name this queue something which notes both the queue it will be linked with, and the fact that it is a Dead Letter Queue.
    For example, "MyLogsQueue-DLQ" if your original SQS queue is named "MyLogsQueue."

Do not select “Dead Letter Queue” during this step.

  1. From the Queues screen, select the original queue you created in the Create a Simple Queue Service section above, and click Edit.

    image-20240301-235615.png

  2. Scroll to the Dead-letter queue section and choose Enabled.

    image-20240301-235633.png

  3. Choose the Amazon Resource Name (ARN) of the new queue you just created.

  4. To configure the number of times that a message can be received before being sent to a dead-letter queue, set Maximum receives to a value between 1 and 1,000.

LogRhythm recommends setting this value to 4.

  1. When you finish configuring the dead-letter queue, choose Save.
    After you save the queue, the console displays the Details page for your queue. On the Details page, the Dead-letter queue tab displays the Maximum Receives and Dead Letter Queue ARN.

AWS Amazon S3 Collector IAM User Permissions          

Prerequisites

  • The bucket and SQS must be created in advance by the admin user.

  • The bucket and SQS must be in the same region.

  • Event notification to the SQS must be configured by the admin user.

Add Permissions to a User

  1. Login to the AWS account at:

    CODE 
    https://console.aws.amazon.com/

  2. On the top menu bar, click Services and use the search bar to search for IAM.

  3. Select IAM (Manage access to AWS resources).

    image-20240301-235643.png

  4. On the side menu, click Users, and in the right pane, click Create user.

    image-20240301-235653.png

  5. Enter the name of the user you want to create in text bar.

    image-20240301-235704.png

  6. Click Next.

  7. Select Attach policies directly.

  8. Use the search bar to search for s3read and select AmazonS3ReadOnlyAccess.

    image-20240301-235714.png

  9. Click Create policy.

  10. On the Visual editor tab, select SQS from the Service drop-down menu.

  11. Under the Access level header, expand the Read drop-down and select GetQueueUrl, GetQueueAttributes, and ReceiveMessage.

    image-20240301-235724.png

  12. Expand the Write drop-down and select DeleteMessage.

    image-20240301-235733.png

  13. Expand the Resources drop-down menu and select Specific, then click Add ARNs.

    image-20240301-235742.png


    The Add ARN(s) dialog box displays.

  14. Provide the Resource region and Resource queue name, and then click Add ARNs.

    image-20240301-235751.png

  15. Click Next.

    image-20240301-235801.png

  16. Provide the name of the policy in the Name field, and then click Create policy.

    image-20240301-235811.png


    The policy is successfully created.

    image-20240301-235825.png

Embed the Policy in the User IAM Permissions

Perform these steps immediately after completing the section above.

  1. Click the Refresh icon in the top-right corner.

  2. Select the newly created policy from the search box.

  3. Click Next.

    The User details and Permissions summary display in the Review section.

  4. Click Create user.

    image-20240301-235839.png


    A user is successfully created using the policy details.

    image-20240301-235849.png

  5. Click on the View User button, and then click on Security credentials.

    image-20240301-235859.png

  6. Under the Access keys header, click on Create access key.

    image-20240301-235907.png

  7. Under Access key best practices, select Other and then click Next.

    image-20240301-235915.png

  8. Click Create access key.

    image-20240301-235923.png

  9. Download the user credentials by clicking Download .csv file, and then click Done.

    image-20240301-235934.png

This file contains the Access Key ID and Secret Access Key required to Configure the Amazon S3 Collector in Axon.

S3 Object Naming Guidelines

It is highly recommended that while creating objects (such as buckets, folder files, etc.) we should follow the guidelines given for naming objects here in the official Amazon documentation.

This document provides a list of safe characters as well as a list of characters to avoid.

Safe Characters

The following character sets are generally safe for use in key names.

Alphanumeric Characters

  • 0-9

  • a-z

  • A-Z

Special Characters

  • Exclamation point (“!”)

  • Hyphen (“-”)

  • Underscore (“_”)

  • Period (“.”)

  • Asterisk (“*”)

  • Single quote (“'“)

  • Open parenthesis (“(“)

  • Close parenthesis (“)”)

  • Ampersand ("&")

  • Dollar ("$")

  • At symbol ("@")

  • Equals ("=")

  • Semicolon (";")

  • Forward slash ("/")

  • Colon (":")

  • Plus ("+")

  • Space (“ “)

Significant sequences of spaces might be lost in some uses (especially multiple spaces).

  • Comma (",")

  • Question mark ("?")

Characters to Avoid 

Try not to use the following characters in a key name because of significant special character handling, which isn't consistent across all applications.

  • Backslash ("\")

  • Left curly brace ("{")

  • Non-printable ASCII characters (128–255 decimal characters)

  • Caret ("^")

  • Right curly brace ("}")

  • Percent character ("%")

  • Grave accent/backtick ("`")

  • Right square bracket ("]")

  • Quotation marks

  • Greater than symbol (">")

  • Left square bracket ("[")

  • Tilde ("~")

  • Less than symbol ("<")

  • Pound character ("#")

  • Vertical bar/pipe ("|")

JavaScript errors detected

Please note, these errors can depend on your browser setup.

If this problem persists, please contact our support.

Contact Support Close