Configure Open Collector Advanced Properties

How Should I Configure Concurrency and Capacity in Advanced Properties?

Configuring these settings for advanced properties may require testing several different configuration settings in your deployment to find the optimum performance. Below is guidance for configuring both concurrency and capacity.

Concurrency

When configuring concurrency, you must consider both channel length and CPU utilization.

If you have set your concurrency to the recommended settings but still do not see optimal MPS, view your channel lengths. The graph may display counters that are plateauing (or "flat-topping"). If so, adjust the concurrency for that component.

In the image below, channel lengths for oc.demux.chan_rcvr.* and oc.pipeline.eventhub.* are "flat-topped." This indicates sub-optimal performance and suggests that a bottleneck exists at the demux and pipeline components (the second prefix of the counter name).

In the following image, concurrency for demux and pipelines has been doubled. Performance improves, but still plateaus:

In this case, double concurrency until "flat-topping" stops:

In the image above, channel lengths no longer plateau, but performance gain is marginal. This may be because system resources have been exhausted, as verified in the image of CPU utilization below:

When CPU is 70% or more, resource constriction means there is not much you can do to improve performance beyond current levels. However, if CPU were only at 20%, you could still configure the settings further.

Note that different log sources require more parsing than others. You may be ingesting a log source that requires more parsing than others, which decreases throughput.

Capacity

When ingesting AWS CloudTrail or other log sources with multiple outputs per input, configuring capacity to a higher value can marginally improve performance. There is no standard value that capacity should be set to. It is recommended that the more memory your system has, the higher capacity should be set.