Configure Azure Event Hub

Azure Event Hubs is a data streaming platform and event ingestion service. It can receive and process millions of events per second. Data sent to an event hub can be transformed and stored by using any real-time analytics provider or batching/storage adapters.

This section details the configuration steps to set up a new Azure Event Hub cloud collector in the Azure portal.

Prerequisites

• Azure portal login credentials and admin access.

Vendor Documentation

• https://learn.microsoft.com/en-us/azure/event-hubs/
• https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-faq
• https://learn.microsoft.com/en-us/azure/event-hubs/event-hubs-create

Create a New Event Hub Namespace

1. Navigate to https://portal.azure.com and log in to the Azure portal.
2. Click All services in the left-hand menu.
3. Click Event Hubs.
   
4. Click the Create button.
   

   

   
   
5. Select your organization's Subscription and Resource group.
   

   

   
   
6. Enter a unique Namespace name.
7. Select the Location from the drop-list.

8. The Pricing tier and Throughput Units must be configured according to your organization's Event Hub subscription.
   

   

   For information on Azure Event Hubs quotas, limits, and pricing tiers, see https://learn.microsoft.com/en-us/azure/event-hubs/event-hubs-quotas

9. Click the Review + Create button.
10. Review the entered information, and then click Create.
    The Validation Succeeded pop-up appears and the namespace is successfully created.
    

    

    
    

Create a New Event Hub on the Namespace

1. On the Event Hubs page, select the created Namespace to open an Overview page.
2. Click the + Event Hub button at the top-left.
   

   

   
   
3. Enter a unique Name for the Event Hub.
   

   

   
   

4. Configure the Partition Count and Message Retention settings per your organization's standards.
   

   

   For more information on partitions, see https://learn.microsoft.com/en-us/azure/event-hubs/event-hubs-scalability#partitions

   For more information on message retention, see https://learn.microsoft.com/en-us/azure/event-hubs/event-hubs-features#event-retention

   

   Message retention is not available for Namespaces on the Basic pricing tier.

5. Click the Review + Create button.
6. Review the entered information, and then click Create.
   The Event Hub is created and appears in the list of event hubs.

7. Verify the status of the newly created Event Hub within Alerts.
   

Create a Consumer Group on the Event Hub

Only one Azure Event Hub Collector should be configured per combination of Consumer Group + Azure Event Hub. Creating multiple Azure Collectors for a single consumer group can result in errors and loss of logs.

Additionally, it is not recommended to change the consumer group on an already-configured Azure Event Hub Collector. This will cause the Collector to re-establish its connection to Azure, resulting in further loss of logs. In cases where the consumer group does need to be changed, retire the old collector and create a new one with the correct consumer group.

1. On the Namespace's Overview page, click Event Hubs in the left-hand panel.
   

   

   
   
2. Select the Event Hub created above.
   
3. Click the + Consumer group button.
   

   

   
   
4. Enter a unique Name for the consumer group.
5. Click the Create button.
   The consumer group is successfully created.

Add Access Policies to the Event Hub

1. On the Namespace's Overview page, click Event Hubs in the left-hand panel.
2. Select the Event Hub created above.
3. Click Shared access policies in the left-hand panel.
   

   

   
   
4. Click Add.
   

   

   
   
5. Enter a unique Policy Name.
   

   

   
   
6. Check the Manage box.
7. Click Create.
   The new access policy is successfully created.
8. Click the name of the newly-created access policy in the list.
9. Copy the Primary key into a text document for later use.

Configure Diagnostics Settings on the Event Hub Namespace

1. On the Namespace's Overview page, click Activity Log in the left-hand panel.
2. Click Diagnostics settings.
   

   

   
   
3. Click Add diagnostic setting.
   

   

   
   
4. Check the following log source types to include in the diagnostic settings:
   1. Administrative
   2. Security
   3. ServiceHealth
   4. Alert
   5. Recommendation
   6. Policy
   7. Autoscale
   8. ResourceHealth
      

      

5. Check the Stream to an event hub option.
6. Select the Event hub namespace created earlier from the drop-list.

7. Click Save.

   

   The maximum number of diagnostic settings that can be created at one time is five.

Create a Storage Account and Storage Account Container

1. On the Azure portal dashboard, click Storage Account.
   
2. Click the Create button.
   
3. Select the Subscription and Resource group used by your organization.
4. Enter a unique Storage account name.
5. Select the Region from the drop-list.
   

   

   
   
6. Select the Performance and Redundancy settings used by your organization.
7. Click Review + Create.
8. Review the entered information, and then click Create.
   The storage account is created successfully.
9. On the created storage account's overview page, click Containers in the left-hand panel.
10. Click the + Container button.
    

    

    
    
11. Enter a unique Name for the container.
    

    

    
    
12. Set the Access level for the container to Public.
13. Click Create.
    The storage account container is successfully created.

Obtain Connection Strings and Access Keys

1. On the Azure portal dashboard, click Storage Account.
   
2. Select the newly-created storage account from the list.
3. Click Access Keys in the left-hand panel.
   
4. Copy any access key name and the associated connection string for that access key to a text document for later use.

Log Generation

To generate the logs for the Azure Event Hub Collector/Integration, multiple activities can be performed manually from the Azure Portal:

• Create/Delete Event Hub Namespace.

• Create/Delete Event Hub.

• Change the “MESSAGE RETENTION” period on any particular Event Hub.

• Create/Delete Consumer Group

• Add/Delete policies on any Event Hub.

• Change the access of any policy on the Event Hub from Manage to Send/Listen or from Send/Listen to Manage, and then save the policy.
  

  

  This option is used most frequently to generate logs.