Scaling Options for Open Collector and Azure EventHub Beat

There are two ways to scale the EventHub Beat setup horizontally:

Horizontally Scale Each Component on the Same Machine

Scale the Azure EventHub Beat Horizontally

To scale the beat horizontally:

1. Start a new beat instance. 

In this approach, a separate instance of the beat will be launched with its dedicated configuration file

2. Configure more than one EventHub in the beat instance by providing different storage container, EventHub connection string and unique container name.

Scale Open Collector Horizontally

The Open Collector component is the combination of several smaller components, such as demux, lumberjack, pipelines, outputs, etc. Horizontal scaling of Open Collector is possible by scaling each individual component. For more information on scaling Open Collector horizontally, see Configure Open Collector Advanced Properties.

Horizontal Scaling Using a New Open Collector Instance

Using this approach, a new instance of the Open Collector beat setup will be launched to achieve the desired goal. In order to decide the number of instances fit for achieving any goal with Open Collector, see Scaling Open Collector and the AZURE Eventhubbeat