Scaling Open Collector and the AWS S3 Beat
The information on this page is prepared using the Machine and Log Specifications for the AWS S3 Beat.
Scaling Options
If your instance is not performing with the desired message per second (MPS) count, then Open Collector can be scaled vertically and horizontally using the following options.
For more information on the operating requirements for the AWS S3 Beat, see the Log Specifications section of Machine and Log Specifications for the AWS S3 Beat.
Configuration | # |
---|---|
Bucket | 1 |
SQS | 1 |
Beats Instance | 1 |
For these configurations, the beat is streaming logs to a single instance of Open Collector running on the same machine with default advanced settings.
Scaling Horizontally
The beat can be scaled horizontally by increasing the 'queuename:region' combination as shown in the tables below to match with machine specifications. For more information on scaling the AWS S3 Beat horizontally, see Scaling Options for Open Collector and the AWS S3 Beat.
If CPU usage is going above 70% even when using the given scaling and configuration options, then no further horizontal scaling will be possible on a single instance. At this point, it is recommended to launch a new instance of Open Collector and Beats to achieve the desired goal.
Scaling Vertically
The options for vertically scaling the AWS S3 Beat vary depending on the monitoring system used within the beat.
Scaling Open Collector and the AWS S3 Beat for Cisco Umbrella
Open Collector can be scaled vertically with the given specifications and system configuration options to align with the MPS capabilities. If the required MPS is 1700, then the specifications presented in Option 1 below are recommended. In order to achieve ~2500 MPS, use the specifications in Option 2. You may also vertically scale the AWS S3 Beat for Cisco Umbrella to achieve ~4000 MPS using Option 3. Beyond this point, the beat can be scaled further by adding more machines in order to achieve a higher MPS performance.
If at any point your CPU consumption is more than 70%, it is recommended to add a new machine to share the load.
Log File Size | Sample Log File |
---|---|
1 KB |
Option 1 (8CPU, 16GB RAM) OC1 | Option 2 (16CPU, 32GB RAM) OC2 | Option 3 (24CPU, 64GB RAM) OC3 |
---|---|---|
MPS: 1700Scale Beat horizontally
YML
Scale OC horizontally
YML
CPU/RAM Consumption
YML
| MPS: 2583Scale Beat horizontally
YML
Scale OC horizontally
YML
CPU/RAM Consumption
YML
| MPS: 4089Scale Beat horizontally
YML
Scale OC horizontally
YML
CPU/RAM Consumption
YML
|
Scaling Open Collector and the AWS S3 Beat for CloudWatch
Open Collector can be scaled vertically with the given specifications and system configuration options to align with the MPS capabilities. If the required MPS is ~1300, then the specifications presented in Option 1 below are recommended. In order to achieve ~2200 MPS, use the specifications in Option 2. You may also vertically scale the AWS S3 Beat for CloudWatch to achieve ~2800 MPS using Option 3. Beyond this point, the beat can be scaled further by adding more machines in order to achieve a higher MPS performance.
If at any point your CPU consumption is more than 70%, it is recommended to add a new machine to share the load.
Log File Size | Sample Log File |
---|---|
3 KB |
Option 1 (8CPU, 16GB RAM) OC1 | Option 2 (16CPU, 32GB RAM) OC2 | Option 3 (24CPU, 64GB RAM) OC3 |
---|---|---|
MPS: 1370Scale Beat horizontally
YML
Scale OC horizontally
YML
CPU/RAM Consumption
YML
| MPS: 2215Scale Beat horizontally
YML
Scale OC horizontally
YML
CPU/RAM Consumption
YML
| MPS: 2825Scale Beat horizontally
YML
Scale OC horizontally
YML
CPU/RAM Consumption
YML
|
Scaling Open Collector and the AWS S3 Beat for GuardDuty
Open Collector can be scaled vertically with the given specifications and system configuration options to align with the MPS capabilities. If the required MPS is ~550, then the specifications presented in Option 1 below are recommended. In order to achieve ~700 MPS, use the specifications in Option 2. You may also vertically scale the AWS S3 Beat for GuardDuty to achieve ~920 MPS using Option 3. Beyond this point, the beat can be scaled further by adding more machines in order to achieve a higher MPS performance.
If at any point your CPU consumption is more than 70%, it is recommended to add a new machine to share the load.
Log File Size | Sample Log File |
---|---|
36 KB |
Option 1 (8CPU, 16GB RAM) OC1 | Option 2 (16CPU, 32GB RAM) OC2 | Option 3 (24CPU, 64GB RAM) OC3 |
---|---|---|
MPS: 560Scale Beat horizontally
YML
Scale OC horizontally
YML
CPU/RAM Consumption
YML
| MPS: 700Scale Beat horizontally
YML
Scale OC horizontally
YML
CPU/RAM Consumption
YML
| MPS: 927Scale Beat horizontally
YML
Scale OC horizontally
YML
CPU/RAM Consumption
YML
|
Scaling Open Collector and the AWS S3 Beat for Config Events
Open Collector can be scaled vertically with the given specifications and system configuration options to align with the MPS capabilities. If the required MPS is ~350, then the specifications presented in Option 1 below are recommended. In order to achieve ~600 MPS, use the specifications in Option 2. You may also vertically scale the AWS S3 Beat for Config Events to achieve ~610 MPS using Option 3. Beyond this point, the beat can be scaled further by adding more machines in order to achieve a higher MPS performance.
If at any point your CPU consumption is more than 70%, it is recommended to add a new machine to share the load.
Log File Size | Sample Log File |
---|---|
12 KB |
Option 1 (8CPU, 16GB RAM) OC1 | Option 2 (16CPU, 32GB RAM) OC2 | Option 3 (24CPU, 64GB RAM) OC3 |
---|---|---|
MPS: 390Scale Beat horizontally
YML
Scale OC horizontally
YML
CPU/RAM Consumption
YML
| MPS: 602Scale Beat horizontally
YML
Scale OC horizontally
YML
CPU/RAM Consumption
YML
| MPS: 610Scale Beat horizontally
YML
Scale OC horizontally
YML
CPU/RAM Consumption
YML
|
Scaling Open Collector and the AWS S3 Beat for CloudTrail
Open Collector can be scaled vertically with the given specifications and system configuration options to align with the MPS capabilities. In the table below, the OC1 and OC2 configurations are flat-topping (maxing out) the channel length, and CPU usage is at 98%; therefore, these machines are not suitable for CloudTrail log collection. For the OC3 configuration, flat-topping from channel length is no longer present, but the CPU usage is at 95%; therefore, scale OC3 vertically to reduce CPU usage to around 70%, and then scale horizontally by adding more machines to achieve a higher MPS performance.
If at any point the Channel Length in the Grafana Dashboard shows flat-topping, or CPU consumption is more than 70%, then it is recommended to scale by adding a new machine to share the load.
Log File Size | Sample Log File |
---|---|
1038KB |
Option 1 (8CPU, 16GB RAM) OC1 | Option 2 (16CPU, 32GB RAM) OC2 | Option 3 (24CPU, 64GB RAM) OC3 |
---|---|---|
MPS: 1002This MPS value for CloudTrail logs is subject to change, and could be improved in the future. Scale Beat horizontally
YML
Scale OC horizontally
YML
CPU/RAM Consumption
YML
As channel length flat tops and CPU usage is at 98%, no further enhancement is possible using this option. Scale this machine vertically. | MPS: 1068This MPS value for CloudTrail logs is subject to change, and could be improved in the future. Scale Beat horizontally
YML
Scale OC horizontally
YML
CPU/RAM Consumption
YML
As channel length flat tops and CPU usage is at 98%, no further enhancement is possible using this option. Scale this machine vertically. | MPS: 1079This MPS value for CloudTrail logs is subject to change, and could be improved in the future. Scale Beat horizontally
YML
Scale OC horizontally
YML
CPU/RAM Consumption
YML
Channel length flat tops are no longer present using these settings, but CPU usage is at 95%; therefore, no further enhancement is possible. Scale this machine vertically to achieve better performance. |