Skip to main content
Skip table of contents

Scaling Open Collector and the AWS S3 Beat


The information on this page is prepared using the Machine and Log Specifications for the AWS S3 Beat.

Scaling Options

If your instance is not performing with the desired message per second (MPS) count, then Open Collector can be scaled vertically and horizontally using the following options.

For more information on the operating requirements for the AWS S3 Beat, see the Log Specifications section of Machine and Log Specifications for the AWS S3 Beat.

Configuration#
Bucket1
SQS1
Beats Instance1

For these configurations, the beat is streaming logs to a single instance of Open Collector running on the same machine with default advanced settings.

Scaling Horizontally

The beat can be scaled horizontally by increasing the 'queuename:region' combination as shown in the tables below to match with machine specifications. For more information on scaling the AWS S3 Beat horizontally, see Scaling Options for Open Collector and the AWS S3 Beat.

If CPU usage is going above 70% even when using the given scaling and configuration options, then no further horizontal scaling will be possible on a single instance. At this point, it is recommended to launch a new instance of Open Collector and Beats to achieve the desired goal.

Scaling Vertically

The options for vertically scaling the AWS S3 Beat vary depending on the monitoring system used within the beat.

Scaling Open Collector and the AWS S3 Beat for Cisco Umbrella 

Open Collector can be scaled vertically with the given specifications and system configuration options to align with the MPS capabilities. If the required MPS is 1700, then the specifications presented in Option 1 below are recommended. In order to achieve ~2500 MPS, use the specifications in Option 2. You may also vertically scale the AWS S3 Beat for Cisco Umbrella to achieve ~4000 MPS using Option 3. Beyond this point, the beat can be scaled further by adding more machines in order to achieve a higher MPS performance.

If at any point your CPU consumption is more than 70%, it is recommended to add a new machine to share the load.

Log File SizeSample Log File
1 KB


Option 1 (8CPU, 16GB RAM) OC1

Option 2 (16CPU, 32GB RAM) OC2

Option 3 (24CPU, 64GB RAM) OC3

MPS: 1700

Scale Beat horizontally

YML
'queuename:region' combinations: 30

Scale OC horizontally

YML
advanced:
  demux:
    capacity: 1000
    concurrency: 10
  lumberjack:
    capacity: 100
    concurrency: 10
  pipelines:
    capacity: 1000
    concurrency: 20
  outputs:
    capacity: 1000
    concurrency: 2
  phonehome:
    capacity: 1000
    concurrency: 1

CPU/RAM Consumption

YML
RAM: 4.42GB
CPU: 80%

MPS: 2583

Scale Beat horizontally

YML
'queuename:region' combinations: 40

Scale OC horizontally

YML
advanced:
  demux:
    capacity: 1000
    concurrency: 20
  lumberjack:
    capacity: 1000
    concurrency: 20
  pipelines:
    capacity: 1000
    concurrency: 30
  outputs:
    capacity: 1000
    concurrency: 4
  phonehome:
    capacity: 1000
    concurrency: 1

CPU/RAM Consumption

YML
RAM: 9.09GB
CPU: 82%

MPS: 4089

Scale Beat horizontally

YML
'queuename:region' combinations: 70

Scale OC horizontally

YML
advanced:
  demux:
    capacity: 1000
    concurrency: 5
  lumberjack:
    capacity: 100
    concurrency: 5
  pipelines:
    capacity: 100
    concurrency: 10
  outputs:
    capacity: 100
    concurrency: 1
  phonehome:
    capacity: 1000
    concurrency: 1

CPU/RAM Consumption

YML
RAM: 2.57GB
CPU: 82%

Scaling Open Collector and the AWS S3 Beat for CloudWatch

Open Collector can be scaled vertically with the given specifications and system configuration options to align with the MPS capabilities. If the required MPS is ~1300, then the specifications presented in Option 1 below are recommended. In order to achieve ~2200 MPS, use the specifications in Option 2. You may also vertically scale the AWS S3 Beat for CloudWatch to achieve ~2800 MPS using Option 3. Beyond this point, the beat can be scaled further by adding more machines in order to achieve a higher MPS performance. 

If at any point your CPU consumption is more than 70%, it is recommended to add a new machine to share the load.

Log File SizeSample Log File
3 KB


Option 1 (8CPU, 16GB RAM) OC1

Option 2 (16CPU, 32GB RAM) OC2


Option 3 (24CPU, 64GB RAM) OC3

MPS: 1370

Scale Beat horizontally

YML
'queuename:region' combinations: 6

Scale OC horizontally

YML
advanced:
  demux:
    capacity: 1000
    concurrency: 10
  lumberjack:
    capacity: 100
    concurrency: 5
  pipelines:
    capacity: 100
    concurrency: 10
  outputs:
    capacity: 100
    concurrency: 5
  phonehome:
    capacity: 1000
    concurrency: 1

CPU/RAM Consumption

YML
RAM: 2.21GB
CPU: 78%

MPS: 2215

Scale Beat horizontally

YML
'queuename:region' combinations: 10

Scale OC horizontally

YML
advanced:
  demux:
    capacity: 1000
    concurrency: 10
  lumberjack:
    capacity: 100
    concurrency: 5
  pipelines:
    capacity: 100
    concurrency: 10
  outputs:
    capacity: 1000
    concurrency: 5
  phonehome:
    capacity: 1000
    concurrency: 1

CPU/RAM Consumption

YML
RAM: 2.22GB
CPU: 75%

MPS: 2825

Scale Beat horizontally

YML
'queuename:region' combinations: 13

Scale OC horizontally

YML
advanced:
  demux:
    capacity: 1000
    concurrency: 10
  lumberjack:
    capacity: 100
    concurrency: 5
  pipelines:
    capacity: 100
    concurrency: 15
  outputs:
    capacity: 100
    concurrency: 5
  phonehome:
    capacity: 1000
    concurrency: 1

CPU/RAM Consumption

YML
RAM: 4.42GB
CPU: 80%

Scaling Open Collector and the AWS S3 Beat for GuardDuty 

Open Collector can be scaled vertically with the given specifications and system configuration options to align with the MPS capabilities. If the required MPS is ~550, then the specifications presented in Option 1 below are recommended. In order to achieve ~700 MPS, use the specifications in Option 2. You may also vertically scale the AWS S3 Beat for GuardDuty to achieve ~920 MPS using Option 3. Beyond this point, the beat can be scaled further by adding more machines in order to achieve a higher MPS performance. 

If at any point your CPU consumption is more than 70%, it is recommended to add a new machine to share the load.

Log File SizeSample Log File
36 KB


Option 1 (8CPU, 16GB RAM) OC1

Option 2 (16CPU, 32GB RAM) OC2

Option 3 (24CPU, 64GB RAM) OC3

MPS: 560

Scale Beat horizontally

YML
'queuename:region' combinations: 12

Scale OC horizontally

YML
advanced:
  demux:
    capacity: 1000
    concurrency: 10
  lumberjack:
    capacity: 100
    concurrency: 5
  pipelines:
    capacity: 100
    concurrency: 10
  outputs:
    capacity: 100
    concurrency: 2
  phonehome:
    capacity: 1000
    concurrency: 1

CPU/RAM Consumption

YML
RAM: 2.02GB
CPU: 88%

MPS: 700

Scale Beat horizontally

YML
'queuename:region' combinations: 15

Scale OC horizontally

YML
advanced:
  demux:
    capacity: 1000
    concurrency: 10
  lumberjack:
    capacity: 100
    concurrency: 5
  pipelines:
    capacity: 100
    concurrency: 10
  outputs:
    capacity: 100
    concurrency: 5
  phonehome:
    capacity: 1000
    concurrency: 1

CPU/RAM Consumption

YML
RAM: 2.12GB
CPU: 78%

MPS: 927

Scale Beat horizontally

YML
'queuename:region' combinations: 20

Scale OC horizontally

YML
advanced:
  demux:
    capacity: 1000
    concurrency: 10
  lumberjack:
    capacity: 100
    concurrency: 5
  pipelines:
    capacity: 100
    concurrency: 10
  outputs:
    capacity: 100
    concurrency: 5
  phonehome:
    capacity: 1000
    concurrency: 1

CPU/RAM Consumption

YML
RAM: 2.38GB
CPU: 72%

Scaling Open Collector and the AWS S3 Beat for Config Events 

Open Collector can be scaled vertically with the given specifications and system configuration options to align with the MPS capabilities. If the required MPS is ~350, then the specifications presented in Option 1 below are recommended. In order to achieve ~600 MPS, use the specifications in Option 2. You may also vertically scale the AWS S3 Beat for Config Events to achieve ~610 MPS using Option 3. Beyond this point, the beat can be scaled further by adding more machines in order to achieve a higher MPS performance. 

If at any point your CPU consumption is more than 70%, it is recommended to add a new machine to share the load.

Log File SizeSample Log File
12 KB


Option 1  (8CPU, 16GB RAM) OC1

Option 2 (16CPU, 32GB RAM) OC2

Option 3 (24CPU, 64GB RAM) OC3

MPS: 390

Scale Beat horizontally

YML
'queuename:region' combinations: 90

Scale OC horizontally

YML
advanced:
  demux:
    capacity: 1000
    concurrency: 5
  lumberjack:
    capacity: 100
    concurrency: 5
  pipelines:
    capacity: 100
    concurrency: 10
  outputs:
    capacity: 100
    concurrency: 1
  phonehome:
    capacity: 1000
    concurrency: 1

CPU/RAM Consumption

YML
RAM: 2.15GB
CPU: 86%

MPS: 602

Scale Beat horizontally

YML
'queuename:region' combinations: 110

Scale OC horizontally

YML
advanced:
  demux:
    capacity: 1000
    concurrency: 5
  lumberjack:
    capacity: 100
    concurrency: 5
  pipelines:
    capacity: 100
    concurrency: 10
  outputs:
    capacity: 100
    concurrency: 1
  phonehome:
    capacity: 1000
    concurrency: 1

CPU/RAM Consumption

YML
RAM: 2.80GB
CPU: 82%

MPS: 610

Scale Beat horizontally

YML
'queuename:region' combinations: 110

Scale OC horizontally

YML
advanced:
  demux:
    capacity: 1000
    concurrency: 5
  lumberjack:
    capacity: 100
    concurrency: 5
  pipelines:
    capacity: 100
    concurrency: 10
  outputs:
    capacity: 100
    concurrency: 1
  phonehome:
    capacity: 1000
    concurrency: 1

CPU/RAM Consumption

YML
RAM: 3.00GB
CPU: 80%

Scaling Open Collector and the AWS S3 Beat for CloudTrail 

Open Collector can be scaled vertically with the given specifications and system configuration options to align with the MPS capabilities. In the table below, the OC1 and OC2 configurations are flat-topping (maxing out) the channel length, and CPU usage is at 98%; therefore, these machines are not suitable for CloudTrail log collection. For the OC3 configuration, flat-topping from channel length is no longer present, but the CPU usage is at 95%; therefore, scale OC3 vertically to reduce CPU usage to around 70%, and then scale horizontally by adding more machines to achieve a higher MPS performance.

If at any point the Channel Length in the Grafana Dashboard shows flat-topping, or CPU consumption is more than 70%, then it is recommended to scale by adding a new machine to share the load.

Log File SizeSample Log File
1038KB


Option 1 (8CPU, 16GB RAM) OC1

Option 2 (16CPU, 32GB RAM) OC2

Option 3  (24CPU, 64GB RAM) OC3

MPS: 1002

This MPS value for CloudTrail logs is subject to change, and could be improved in the future.

Scale Beat horizontally

YML
'queuename:region' combinations: 1

Scale OC horizontally

YML
advanced:
  demux:
    capacity: 1000
    concurrency: 5
  lumberjack:
    capacity: 100
    concurrency: 5
  pipelines:
    capacity: 100
    concurrency: 10
  outputs:
    capacity: 100
    concurrency: 1
  phonehome:
    capacity: 1000
    concurrency: 1

CPU/RAM Consumption

YML
RAM: 4.42GB
CPU: 98%

As channel length flat tops and CPU usage is at 98%, no further enhancement is possible using this option. Scale this machine vertically.

MPS: 1068

This MPS value for CloudTrail logs is subject to change, and could be improved in the future.

Scale Beat horizontally

YML
'queuename:region' combinations: 1

Scale OC horizontally

YML
advanced:
  demux:
    capacity: 1000
    concurrency: 10
  lumberjack:
    capacity: 100
    concurrency: 10
  pipelines:
    capacity: 2000
    concurrency: 40
  outputs:
    capacity: 1000
    concurrency: 10
  phonehome:
    capacity: 1000
    concurrency: 1

CPU/RAM Consumption

YML
RAM: 6.84GB
CPU: 98%

As channel length flat tops and CPU usage is at 98%, no further enhancement is possible using this option. Scale this machine vertically.

MPS: 1079

This MPS value for CloudTrail logs is subject to change, and could be improved in the future.

Scale Beat horizontally

YML
'queuename:region' combinations: 1

Scale OC horizontally

YML
advanced:
  demux:
    capacity: 1000
    concurrency: 40
  lumberjack:
    capacity: 1000
    concurrency: 10
  pipelines:
    capacity: 2000
    concurrency: 50
  outputs:
    capacity: 1000
    concurrency: 20
  phonehome:
    capacity: 1000
    concurrency: 1

CPU/RAM Consumption

YML
RAM: 9.26GB
CPU: 95%

Channel length flat tops are no longer present using these settings, but CPU usage is at 95%; therefore, no further enhancement is possible. Scale this machine vertically to achieve better performance.

JavaScript errors detected

Please note, these errors can depend on your browser setup.

If this problem persists, please contact our support.