Scaling Open Collector and the AWS S3 Beat

The information on this page is prepared using the Machine and Log Specifications for the AWS S3 Beat.

Scaling Options

If your instance is not performing with the desired message per second (MPS) count, then Open Collector can be scaled vertically and horizontally using the following options.

For more information on the operating requirements for the AWS S3 Beat, see the Log Specifications section of Machine and Log Specifications for the AWS S3 Beat.

Configuration	#
Bucket	1
SQS	1
Beats Instance	1

For these configurations, the beat is streaming logs to a single instance of Open Collector running on the same machine with default advanced settings.

Scaling Horizontally

The beat can be scaled horizontally by increasing the 'queuename:region' combination as shown in the tables below to match with machine specifications. For more information on scaling the AWS S3 Beat horizontally, see Scaling Options for Open Collector and the AWS S3 Beat.

If CPU usage is going above 70% even when using the given scaling and configuration options, then no further horizontal scaling will be possible on a single instance. At this point, it is recommended to launch a new instance of Open Collector and Beats to achieve the desired goal.

Scaling Vertically

The options for vertically scaling the AWS S3 Beat vary depending on the monitoring system used within the beat.

Scaling Open Collector and the AWS S3 Beat for Cisco Umbrella

Open Collector can be scaled vertically with the given specifications and system configuration options to align with the MPS capabilities. If the required MPS is 1700, then the specifications presented in Option 1 below are recommended. In order to achieve ~2500 MPS, use the specifications in Option 2. You may also vertically scale the AWS S3 Beat for Cisco Umbrella to achieve ~4000 MPS using Option 3. Beyond this point, the beat can be scaled further by adding more machines in order to achieve a higher MPS performance.

If at any point your CPU consumption is more than 70%, it is recommended to add a new machine to share the load.

Log File Size	Sample Log File
1 KB	cisco_umb.gz

Option 1 (8CPU, 16GB RAM) OC1

Option 2 (16CPU, 32GB RAM) OC2

Option 3 (24CPU, 64GB RAM) OC3

MPS: 1700

Scale Beat horizontally

YML

'queuename:region' combinations: 30

Scale OC horizontally

YML

advanced:
  demux:
    capacity: 1000
    concurrency: 10
  lumberjack:
    capacity: 100
    concurrency: 10
  pipelines:
    capacity: 1000
    concurrency: 20
  outputs:
    capacity: 1000
    concurrency: 2
  phonehome:
    capacity: 1000
    concurrency: 1

CPU/RAM Consumption

YML

RAM: 4.42GB
CPU: 80%

MPS: 2583

Scale Beat horizontally

YML

'queuename:region' combinations: 40

Scale OC horizontally

YML

advanced:
  demux:
    capacity: 1000
    concurrency: 20
  lumberjack:
    capacity: 1000
    concurrency: 20
  pipelines:
    capacity: 1000
    concurrency: 30
  outputs:
    capacity: 1000
    concurrency: 4
  phonehome:
    capacity: 1000
    concurrency: 1

CPU/RAM Consumption

YML

RAM: 9.09GB
CPU: 82%

MPS: 4089

Scale Beat horizontally

YML

'queuename:region' combinations: 70

Scale OC horizontally

YML

advanced:
  demux:
    capacity: 1000
    concurrency: 5
  lumberjack:
    capacity: 100
    concurrency: 5
  pipelines:
    capacity: 100
    concurrency: 10
  outputs:
    capacity: 100
    concurrency: 1
  phonehome:
    capacity: 1000
    concurrency: 1

CPU/RAM Consumption

YML

RAM: 2.57GB
CPU: 82%

Scaling Open Collector and the AWS S3 Beat for CloudWatch

Open Collector can be scaled vertically with the given specifications and system configuration options to align with the MPS capabilities. If the required MPS is ~1300, then the specifications presented in Option 1 below are recommended. In order to achieve ~2200 MPS, use the specifications in Option 2. You may also vertically scale the AWS S3 Beat for CloudWatch to achieve ~2800 MPS using Option 3. Beyond this point, the beat can be scaled further by adding more machines in order to achieve a higher MPS performance.

If at any point your CPU consumption is more than 70%, it is recommended to add a new machine to share the load.

Log File Size	Sample Log File
3 KB	cloud_watch_logs.gz

Option 1 (8CPU, 16GB RAM) OC1

Option 2 (16CPU, 32GB RAM) OC2

Option 3 (24CPU, 64GB RAM) OC3

MPS: 1370

Scale Beat horizontally

YML

'queuename:region' combinations: 6

Scale OC horizontally

YML

advanced:
  demux:
    capacity: 1000
    concurrency: 10
  lumberjack:
    capacity: 100
    concurrency: 5
  pipelines:
    capacity: 100
    concurrency: 10
  outputs:
    capacity: 100
    concurrency: 5
  phonehome:
    capacity: 1000
    concurrency: 1

CPU/RAM Consumption

YML

RAM: 2.21GB
CPU: 78%

MPS: 2215

Scale Beat horizontally

YML

'queuename:region' combinations: 10

Scale OC horizontally

YML

advanced:
  demux:
    capacity: 1000
    concurrency: 10
  lumberjack:
    capacity: 100
    concurrency: 5
  pipelines:
    capacity: 100
    concurrency: 10
  outputs:
    capacity: 1000
    concurrency: 5
  phonehome:
    capacity: 1000
    concurrency: 1

CPU/RAM Consumption

YML

RAM: 2.22GB
CPU: 75%

MPS: 2825

Scale Beat horizontally

YML

'queuename:region' combinations: 13

Scale OC horizontally

YML

advanced:
  demux:
    capacity: 1000
    concurrency: 10
  lumberjack:
    capacity: 100
    concurrency: 5
  pipelines:
    capacity: 100
    concurrency: 15
  outputs:
    capacity: 100
    concurrency: 5
  phonehome:
    capacity: 1000
    concurrency: 1

CPU/RAM Consumption

YML

RAM: 4.42GB
CPU: 80%

Scaling Open Collector and the AWS S3 Beat for GuardDuty

Open Collector can be scaled vertically with the given specifications and system configuration options to align with the MPS capabilities. If the required MPS is ~550, then the specifications presented in Option 1 below are recommended. In order to achieve ~700 MPS, use the specifications in Option 2. You may also vertically scale the AWS S3 Beat for GuardDuty to achieve ~920 MPS using Option 3. Beyond this point, the beat can be scaled further by adding more machines in order to achieve a higher MPS performance.

If at any point your CPU consumption is more than 70%, it is recommended to add a new machine to share the load.

Log File Size	Sample Log File
36 KB	gauradduty.json

Option 1 (8CPU, 16GB RAM) OC1

Option 2 (16CPU, 32GB RAM) OC2

Option 3 (24CPU, 64GB RAM) OC3

MPS: 560

Scale Beat horizontally

YML

'queuename:region' combinations: 12

Scale OC horizontally

YML

advanced:
  demux:
    capacity: 1000
    concurrency: 10
  lumberjack:
    capacity: 100
    concurrency: 5
  pipelines:
    capacity: 100
    concurrency: 10
  outputs:
    capacity: 100
    concurrency: 2
  phonehome:
    capacity: 1000
    concurrency: 1

CPU/RAM Consumption

YML

RAM: 2.02GB
CPU: 88%

MPS: 700

Scale Beat horizontally

YML

'queuename:region' combinations: 15

Scale OC horizontally

YML

advanced:
  demux:
    capacity: 1000
    concurrency: 10
  lumberjack:
    capacity: 100
    concurrency: 5
  pipelines:
    capacity: 100
    concurrency: 10
  outputs:
    capacity: 100
    concurrency: 5
  phonehome:
    capacity: 1000
    concurrency: 1

CPU/RAM Consumption

YML

RAM: 2.12GB
CPU: 78%

MPS: 927

Scale Beat horizontally

YML

'queuename:region' combinations: 20

Scale OC horizontally

YML

advanced:
  demux:
    capacity: 1000
    concurrency: 10
  lumberjack:
    capacity: 100
    concurrency: 5
  pipelines:
    capacity: 100
    concurrency: 10
  outputs:
    capacity: 100
    concurrency: 5
  phonehome:
    capacity: 1000
    concurrency: 1

CPU/RAM Consumption

YML

RAM: 2.38GB
CPU: 72%

Scaling Open Collector and the AWS S3 Beat for Config Events

Open Collector can be scaled vertically with the given specifications and system configuration options to align with the MPS capabilities. If the required MPS is ~350, then the specifications presented in Option 1 below are recommended. In order to achieve ~600 MPS, use the specifications in Option 2. You may also vertically scale the AWS S3 Beat for Config Events to achieve ~610 MPS using Option 3. Beyond this point, the beat can be scaled further by adding more machines in order to achieve a higher MPS performance.

If at any point your CPU consumption is more than 70%, it is recommended to add a new machine to share the load.

Log File Size	Sample Log File
12 KB	config_evnts.json

Option 1 (8CPU, 16GB RAM) OC1

Option 2 (16CPU, 32GB RAM) OC2

Option 3 (24CPU, 64GB RAM) OC3

MPS: 390

Scale Beat horizontally

YML

'queuename:region' combinations: 90

Scale OC horizontally

YML

advanced:
  demux:
    capacity: 1000
    concurrency: 5
  lumberjack:
    capacity: 100
    concurrency: 5
  pipelines:
    capacity: 100
    concurrency: 10
  outputs:
    capacity: 100
    concurrency: 1
  phonehome:
    capacity: 1000
    concurrency: 1

CPU/RAM Consumption

YML

RAM: 2.15GB
CPU: 86%

MPS: 602

Scale Beat horizontally

YML

'queuename:region' combinations: 110

Scale OC horizontally

YML

advanced:
  demux:
    capacity: 1000
    concurrency: 5
  lumberjack:
    capacity: 100
    concurrency: 5
  pipelines:
    capacity: 100
    concurrency: 10
  outputs:
    capacity: 100
    concurrency: 1
  phonehome:
    capacity: 1000
    concurrency: 1

CPU/RAM Consumption

YML

RAM: 2.80GB
CPU: 82%

MPS: 610

Scale Beat horizontally

YML

'queuename:region' combinations: 110

Scale OC horizontally

YML

advanced:
  demux:
    capacity: 1000
    concurrency: 5
  lumberjack:
    capacity: 100
    concurrency: 5
  pipelines:
    capacity: 100
    concurrency: 10
  outputs:
    capacity: 100
    concurrency: 1
  phonehome:
    capacity: 1000
    concurrency: 1

CPU/RAM Consumption

YML

RAM: 3.00GB
CPU: 80%

Scaling Open Collector and the AWS S3 Beat for CloudTrail

Open Collector can be scaled vertically with the given specifications and system configuration options to align with the MPS capabilities. In the table below, the OC1 and OC2 configurations are flat-topping (maxing out) the channel length, and CPU usage is at 98%; therefore, these machines are not suitable for CloudTrail log collection. For the OC3 configuration, flat-topping from channel length is no longer present, but the CPU usage is at 95%; therefore, scale OC3 vertically to reduce CPU usage to around 70%, and then scale horizontally by adding more machines to achieve a higher MPS performance.

If at any point the Channel Length in the Grafana Dashboard shows flat-topping, or CPU consumption is more than 70%, then it is recommended to scale by adding a new machine to share the load.

Log File Size	Sample Log File
1038KB	cloudtrail_log.json

Option 1 (8CPU, 16GB RAM) OC1	Option 2 (16CPU, 32GB RAM) OC2	Option 3 (24CPU, 64GB RAM) OC3
MPS: 1002 This MPS value for CloudTrail logs is subject to change, and could be improved in the future. Scale Beat horizontally YML `'queuename:region' combinations: 1` Scale OC horizontally YML `advanced: demux: capacity: 1000 concurrency: 5 lumberjack: capacity: 100 concurrency: 5 pipelines: capacity: 100 concurrency: 10 outputs: capacity: 100 concurrency: 1 phonehome: capacity: 1000 concurrency: 1` CPU/RAM Consumption YML `RAM: 4.42GB CPU: 98%` As channel length flat tops and CPU usage is at 98%, no further enhancement is possible using this option. Scale this machine vertically.	MPS: 1068 This MPS value for CloudTrail logs is subject to change, and could be improved in the future. Scale Beat horizontally YML `'queuename:region' combinations: 1` Scale OC horizontally YML `advanced: demux: capacity: 1000 concurrency: 10 lumberjack: capacity: 100 concurrency: 10 pipelines: capacity: 2000 concurrency: 40 outputs: capacity: 1000 concurrency: 10 phonehome: capacity: 1000 concurrency: 1` CPU/RAM Consumption YML `RAM: 6.84GB CPU: 98%` As channel length flat tops and CPU usage is at 98%, no further enhancement is possible using this option. Scale this machine vertically.	MPS: 1079 This MPS value for CloudTrail logs is subject to change, and could be improved in the future. Scale Beat horizontally YML `'queuename:region' combinations: 1` Scale OC horizontally YML `advanced: demux: capacity: 1000 concurrency: 40 lumberjack: capacity: 1000 concurrency: 10 pipelines: capacity: 2000 concurrency: 50 outputs: capacity: 1000 concurrency: 20 phonehome: capacity: 1000 concurrency: 1` CPU/RAM Consumption YML `RAM: 9.26GB CPU: 95%` Channel length flat tops are no longer present using these settings, but CPU usage is at 95%; therefore, no further enhancement is possible. Scale this machine vertically to achieve better performance.

Scaling Options

Scaling Horizontally

Scaling Vertically

Scaling Open Collector and the AWS S3 Beat for Cisco Umbrella

Option 1 (8CPU, 16GB RAM) OC1

Option 2 (16CPU, 32GB RAM) OC2

Option 3 (24CPU, 64GB RAM) OC3

MPS: 1700

MPS: 2583

MPS: 4089

Scaling Open Collector and the AWS S3 Beat for CloudWatch

Option 1 (8CPU, 16GB RAM) OC1

Option 2 (16CPU, 32GB RAM) OC2

Option 3 (24CPU, 64GB RAM) OC3

MPS: 1370

MPS: 2215

MPS: 2825

Scaling Open Collector and the AWS S3 Beat for GuardDuty

Option 1 (8CPU, 16GB RAM) OC1

Option 2 (16CPU, 32GB RAM) OC2

Option 3 (24CPU, 64GB RAM) OC3

MPS: 560

MPS: 700

MPS: 927

Scaling Open Collector and the AWS S3 Beat for Config Events

Option 1 (8CPU, 16GB RAM) OC1

Option 2 (16CPU, 32GB RAM) OC2

Option 3 (24CPU, 64GB RAM) OC3

MPS: 390

MPS: 602

MPS: 610

Scaling Open Collector and the AWS S3 Beat for CloudTrail

Option 1 (8CPU, 16GB RAM) OC1

Option 2 (16CPU, 32GB RAM) OC2

Option 3 (24CPU, 64GB RAM) OC3

MPS: 1002

MPS: 1068

MPS: 1079