Scaling Options for Open Collector and AWS S3 Beat

There are two ways to scale the AWS S3 Beat setup horizontally:

Horizontally Scale Each Component on the Same Machine

Scale the AWS S3 Beat Horizontally

To scale the beat horizontally:

Start a new beat instance.
In this approach, a separate instance of the beat will be launched with its dedicated configuration file.

Using this approach, the beat can be scaled to read logs from any set of accounts. Each instance of the beat can read logs from the set of accounts and pass them to a single OC for further processing. However, using this method requires you to provide same set of configurations again for each instance when logs are to be collected from the same account.
(Optional.) Provide extra key-value pairs in a single beat instance.
This is a special case for some the beats (such as the AWS S3 Beat and Eventhub Beat) to limit the scope of the beat.
To provide extra key-value pairs to scale the AWS S3 Beat horizontally, do the following:
1. Enter multiple 'queuename:region' combinations as needed before typing 'c' while configuring a new beat instance:
2. Check the S3 Beat configuration file using following command:
  BASH
```
./lrctl s3beat config view
```
  For example, for five 'queuename:region' combinations, the command above should output the following:

Scale Open Collector Horizontally

The Open Collector component is the combination of several smaller components, such as demux, lumberjack, pipelines, outputs, etc. For more information on each of these components, see Open Collector Advanced Properties. Horizontal scaling of Open Collector is possible by scaling each individual component. For more information on scaling Open Collector horizontally, see Configure Open Collector Advanced Properties.

Horizontal Scaling Using a New Open Collector Instance

Using this approach, a new instance of the Open Collector beat setup will be launched to achieve the desired goal. In order to decide the number of instances fit for achieving any goal with Open Collector, see Scaling Open Collector and the AWS S3 Beat.