Scaling Open Collector and the AZURE Eventhubbeat

The information on this page is prepared using the Machine and Log Specifications for the Azure EventHub Beat.

Scaling Options

If your instance is not performing with the desired message per second (MPS) count, then Open Collector can be scaled vertically and horizontally using the following options.

For these configurations, the beat is streaming logs to a single instance of Open Collector running on the same machine with default advanced settings.

Scaling Horizontally

The beat can be scaled horizontally by configuring more than one EventHub in the EventHub beat instance by providing different storage container, EventHub connection string and unique container name. For more information on scaling the Azure EventHub Beat horizontally, see Scaling Options for Open Collector and the Azure EventHub Beat.

If CPU usage is going above 70% even when using the given scaling and configuration options, then no further horizontal scaling will be possible on a single instance. At this point, it is recommended to launch a new instance of Open Collector and Beats to achieve the desired goal.

Grafana Dashboard Stats:

Note: In case of EventHub beat CPU utilization is not constant, it is higher when it is processing logs with max MPS, CPU and memory utilization manages to come down and gave low output MPS.

Scaling Vertically

The options for vertically scaling the Azure EventHub Beat vary depending on the monitoring system used within the beat.

Scaling Open Collector and the Azure EventHub Beat

Open Collector can be scaled vertically with the given specifications and system configuration options to align with the MPS capabilities. If the required MPS is 2000, then the specification presented below is recommended. Beyond this point, the beat can be scaled further by adding more machines in order to achieve a higher MPS performance with same specification

If at any point your CPU consumption is more than 70%, it is recommended to add a new machine to share the load.