Skip to main content
Skip table of contents

Configure Amazon S3

Overview

  • The Amazon S3 Collector supports log collection from multiple S3 buckets and AWS accounts.

  • Each AWS S3 bucket from which you want to collect logs should be configured to send Object Create Events to a Simple Queue Service (SQS) queue. Then, you provide the queue name(s) and region(s) to the Amazon S3 Collector.

  • The Amazon S3 Collector offers one authentication method: key-based security credentials (access key and secret key).

  • The Amazon S3 Collector supports the use of the Dead Letter Queue service in SQS.  This allows problematic messages to be removed from the SQS queue and stored for later analysis.

Log Source Types

The Amazon S3 Collector collects logs for any file format, but it supports decompression of the .gz file format. Only single-level depth decompression is supported for any file format.

S3 buckets can contain a variety of log source types.  Currently, files in S3 buckets are assumed to contain many single-line logs, delimited by a newline character (\n).

Role-Based Authentication

The Amazon AWS S3 Collector currently does not support role-based authentication.

Access and Configure AWS S3 and SQS

This section provides information for accessing and configuring your AWS S3 instance.

Obtain AWS Access and Secret Keys (Key-Based Authentication)

To obtain AWS access and secret keys, perform the following steps:

  1. Log in to the AWS account.

  2. Click the user profile in the top-right corner, and then click My Security Credentials.

  3. Click Create access key.

  4. Download the .csv file or copy the Access key ID and paste it into a text document.

  5. Copy the Secret access key and paste it into a text document.

Create a Simple Queue Service (SQS) in AWS (Role-Based Authentication)

  1. Log in to the AWS account.

  2. On the top menu bar, click Services.

  3. In the search field, enter sqs, and then select Simple Queue Service from the suggested search results.

  4. In the upper-right corner, click Create queue.

  5. Select Standard from the options. Enter a name for the queue, and then copy and paste it into a text document for use in future steps.

  6. When prompted for Choose method, select Advanced, and then update the policy document.

  7. In your policy document, update the following lines. (Or use the policy generator as defined in step 8.)

    1. Replace SQS-queue-ARN with your SQS ARN. 

    2. Replace awsexamplebucket1 with your bucket name. 

    3. Replace bucket-owner-account-id with your account ID where the bucket resides.

      CODE
      {
      "Version": "2012-10-17",
      "Id": "__default_policy_ID",
      "Statement": [
      {
      "Sid": "__owner_statement",
      "Effect": "Allow",
      "Principal": {
       	"Service": "s3.amazonaws.com"
        },
      "Action": [
      	"sqs:DeleteMessage",
          "sqs:GetQueueUrl",
          "sqs:ReceiveMessage",
          "sqs:SendMessage"
      ],
      "Resource": "SQS-queue-ARN",
      "Condition": {
      	"ArnLike": {
      		"aws:SourceArn": "arn:aws:s3:::awsexamplebucket1"
      	},
      	"StringEquals": {
      		"aws:SourceAccount": "bucket-owner-account-id"
      	}
      }
      }
      ]
      }


  8. Click Policy generator. (Skip this step if you have already generated the policy in step 7.)

  9. Create the Policy:

    1. From the Select Type of Policy drop-down menu, select SQS Queue Policy.


    2. For the Effect option, choose Allow.
    3. In the Principal field, type s3.amazonaws.com

    4. From the Actions drop-down menu, select Send Message, DeleteMessage, GetQueueURL, and ReceiveMessage.
    5. In the Amazon Resource Name (ARN) field, enter your ARN in the following format:
      arn:aws:sqs:<region>:<account_ID>:<queue_name> 

      Replace <region> with your S3 bucket region, <account_ID> with your account ID, and <queue_name> with your queue name.



    6. Click Add conditions.
      • From the Condition drop-down menu, select ArnLike.
      • From the Key drop-down menu, select aws:SourceArn.
      • In the Value field, enter arn:aws:s3:::awsexamplebucket1.

        Replace awsexamplebucket1 with your bucket name.


    7. Click Add Condition.
      • From the Condition drop-down menu, select StringEquals.
      • From the Key drop-down menu, select aws:sourceAccount.
      • In the Value field, enter your account ID.
    8. Click Add Statement.
    9. Click Generate Policy.

      Replace "AWS": [ "s3.amazonaws.com" ] with "Service": [  "s3.amazonaws.com"  ] in the generated policy.

      The Generated policy should look like the following:

      CODE
      {
        "Version": "2012-10-17",
        "Id": "Policy1617965803099",
        "Statement": [
          {
            "Sid": "Stmt1617965799569",
            "Effect": "Allow",
            "Principal": {
              "Service": "s3.amazonaws.com"
            },
            "Action": [
              "sqs:DeleteMessage",
              "sqs:GetQueueUrl",
              "sqs:ReceiveMessage",
              "sqs:SendMessage"
            ],
            "Resource": "arn:aws:sqs:us-east-1:1234567890:test-queue",
            "Condition": {
              "StringEquals": {
                "aws:SourceAccount": "12345678910"
              },
              "ArnLike": {
                "aws:SourceArn": "arn:aws:s3:::test-bucket"
              }
            }
          }
        ]
      }

      The Account ID and other details shown in the above example policy are sample values.

    10. Copy the generated policy. 
    11. Go back to the SQS policy tab, and paste the policy there.
  10. Click Create queue.

    If server-side encryption is enabled on your SQS queue, you will encounter an error similar to this one (with different bucket naming):       

        An error occurred: AnalyticsBucket - Unable to validate the following destination configurations (Service: Amazon S3; Status Code: 400; Error Code: InvalidArgument; Request ID: E2A1F8BD6BEE6EF4;)

    See this AWS documentation to resolve the issue: https://docs.aws.amazon.com/AmazonS3/latest/userguide/grant-destinations-permissions-to-s3.html

Configure Events to Be Sent to SQS Queues

  1. Log in to the AWS account.

  2. On the menu bar at the top, click Services.

  3. In the search bar, enter s3, and then select S3 (Scalable Storage in the Cloud) from the suggested search results.

  4. Search for the bucket from which you want to get events.

  5. Click the name of the bucket, and then click the Properties tab.

  6. Under Advanced settings, click Events.

  7. Click Add notification.


  8. Configure the following permissions:

    1. Name. Enter any name.

    2. Events. Select the events you need to be notified of. For example, All object create events.

    3. Send to. Enter the destination queue (created in the previous procedure) where you want to send events.

    4. SQS. Select the queue created in the previous procedure.

  9. Click Save.

    If you have correctly configured the event, the tile displays active notifications.

  10. Click the Permissions tab to set bucket ownership of objects.
  11. Scroll down and under the Object ownership section, click Edit.
  12. Select Bucket owner preferred and click Save changes.

Create and Configure the SQS Dead Letter Queue

Prerequisites

  • You must have an existing SQS queue, configured via the steps in "Access and Configure AWS S3 and SQS."
  • The dead letter queue and the SQS queue sending logs to LogRhythm must be in the same region.
  • The dead letter queue must be configured as a "standard" queue.

Create Dead Letter Queue and Associate With Existing SQS Queue

  1. Log in to the AWS account.

  2. Open the Amazon SQS console at https://console.aws.amazon.com/sqs/.
    Alternatively, you can click on the top menu bar, click Services and use the search bar to search for SQS.
  3. In the navigation pane, choose Queues.
  4. Click the Create queue button. 
  5. Create a new Standard queue, with default settings.  
    1. Name this queue something which notes both the queue it will be linked with, and the fact that it is a Dead Letter Queue. For example, "MyLogsQueue-DLQ" if your original SQS queue is named "MyLogsQueue."
    2. Do NOT select "Dead Letter Queue" at this time.

      This must be enabled as a dead letter queue from the originating SQS queue.

  6. From the Queues screen, select the original queue you created in the Create a Simple Queue Service section above.

    This should be the queue that you set up in your S3 policy in order to be forwarded to LogRhythm.

  7. Choose Edit.
  8. Scroll to the Dead-letter queue section and choose Enabled.

    Currently, the AWS UI contains a contextual error in this portion. The line "Set this queue to receive undeliverable messages" is confusing. It is important to understand that this feature is being enabled on your original queue, and then selecting the newly created "DLQ" queue as the queue which will receive our Dead Letter messages.

  9. Choose the Amazon Resource Name (ARN) of the new queue you just created.
  10. To configure the number of times that a message can be received before being sent to a dead-letter queue, set Maximum receives to a value between 1 and 1,000.

    LogRhythm recommends setting this value to 4.

  11. When you finish configuring the dead-letter queue, choose Save.
    After you save the queue, the console displays the Details page for your queue. On the Details page, the Dead-letter queue tab displays the Maximum Receives and Dead Letter Queue ARN.

AWS Amazon S3 Collector IAM User Permissions

Prerequisites

  • The bucket and SQS must be created in advance by the admin user.
  • The bucket and SQS must be in the same region.
  • Event notification to the SQS must be configured by the admin user.

Add Permissions to a User

  1. Log in to the AWS account.

  2. On the top menu bar, click Services and use the search bar to search for IAM.

  3. Select IAM (Manage access to AWS resources).

  4. On the side menu, click Users, and in the right pane, click Add user.
  5. Set user details:
    1. Enter the name of the user you want to create in text bar.
    2. Under Access Type, select Programmatic access (you will use the generated access key ID and secret key in the Amazon S3 Collector service).
    3. Click Next: Permissions.
  6. Go to Attach existing policies directly.
    1. Use the search bar to search for s3read and select AmazonS3ReadOnlyAccess.
    2. Click Create policy.
  7. Create the policy:
    1. On the Visual editor tab, select SQS from the Service drop-down menu.
    2. Select GetQueueUrl and ReceiveMessage from the Read drop-down menu under Access level.

    3. Select DeleteMessage from the Write drop-down menu under Access level.
    4. Select Specific and click Add ARN under the Resources drop-down menu.

      The Add ARN(s) dialog box displays.
    5. Provide the Region and then click Add.

    6. Click Next: Tags.

    7. Click Next: Review.

    8. Provide the name of the policy in the Name field, and then click Create policy.

      The policy has been created successfully.

Embed the Policy in the User IAM Permissions

Perform these steps immediately after completing the section above.

  1. Click the Refresh icon in the top-right corner.
  2. Select the newly created policy from the search box.
  3. Click Next: Tags
  4. Click Next: Review.
  5. The User details and Permissions summary display in the Review section. Click Create user.

    A user is successfully created using the policy details.
  6. Download the user credentials by clicking Download .csv.

    You can retrieve the required access key and secret access key to be used in the AWS Amazon S3 Collector from the downloaded csv file.


  7. The user is now shown with the attached policies.



JavaScript errors detected

Please note, these errors can depend on your browser setup.

If this problem persists, please contact our support.