Troubleshoot Azure and Event Hub

A video walk through of configuration is available by going to Documentation & Downloads on the LogRhythm Community and selecting the Open Collector tab.

The following options may help fix common issues:

• If configuring diagnostics generates errors in the Portal, the Activity Log in Azure Monitor often gives a short description of the error. To export the Activity Log to an Event Hub, Azure requires you have a Microsoft Insights Application set up. In Azure Monitor, find the Insights → Applications tag, then add a new Application of Type General.
  

  

• When enabling resource-level diagnostics, the Event Hub must be in the same Region as the Resource. The customer may need to create multiple Event Hub Namespaces, one for each region they have infrastructure in.
• Verify that Azure has been configured correctly from the Azure Portal.
  1. Open the Event Hub Namespace.
  2. In the Show Metrics options, select Messages.
  3. If the Incoming Messages is 0, no logs have been configured from Azure Monitor to send to this Event Hub.
  4. If the Outgoing Messages is 0, the Beat might not be configured.
     

     

     
     
     

Azure portal configuration is owned by Microsoft. This troubleshooting section provides as much information as possible, but LogRhythm ownership of any problems does not begin until the customer has logs sending to Event Hub.

Errors Occur During the Open Collector Event Hub Beat Configuration

• Ensure Azure Connection Strings are in the correct format. For example:

  • Storage Account Connection String:

    CODE

    DefaultEndpointsProtocol=https;AccountName=lrbeatstorage;AccountKey=YyFGcSa4jgHhK/redacted/gnKpJGgMDpa12Ykw==;EndpointSuffix=core.windows.net

  • Event Hub Connection Strings:

    CODE

    Endpoint=sb://lreventhub.servicebus.windows.net/;SharedAccessKeyName=policy5;SharedAccessKey=redacted;EntityPath=insights-operational-logs
    Endpoint=sb://lreventhub.servicebus.windows.net/;SharedAccessKeyName=policy2;SharedAccessKey=redacted;EntityPath=insights-logs-networksecuritygrouprulecounter

• Verify that the Event Hub Connection Strings contain EntityPath=. If they do not, the Shared Access Key came from the Event Hub Namespace, not the Event Hub itself. Replace it with the Event Hub Connection String. For more information, see Configure the Azure Event Hub Using Connection Strings.

• If a configuration is in a bad state, you can reset it and start over.

After Configuration, No Pending Log Source Appears in the Client Console

Use the information in Understand Log Flow From Azure to Event Hub to identify at what point log flow stops.

The best place to start looking is Metrics. Metrics quickly show you if any logs are reaching the Open Collector. If Metrics shows all 0 graphs, check:

• Event Hub metrics from the Azure Portal
• Event Hub logs for errors

Azure Logs Are Making It To LogRhythm But Some Logs Are Missing

• Do the logs originate from Activity Log or Diagnostic Log?

  • Activity Log. Verify they are being sent to an Event Hub.
  • Diagnostic Log. Verify they are turned on.
• Check the Pipeline graphs (especially the Errors graph) in Metrics to ensure all logs are making it through the Open Collector. If you see errors, run the troubleshoot command and open a support ticket so they can escalate the case to engineering, if necessary. It is likely of a result of a log in an unexpected format that our transforms cannot yet handle. The Azure logs that caused the errors are included in the Open Collector log file, which gets packaged by the lrctl troubleshoot command.

StatusCode= 429 Error While Running EventHubBeat

If you you have exhausted your quota limit of API calls to Azure Storage, you will see the following error: Failed to start EPH for Event hub insights-operational-logs : storage.AccountsClient#ListAccountSAS: Failure sending request: StatusCode=429 --". To resolve it, do one of the following:

• Wait, as eventhubbeat will recover on its own within few minutes.
• Raise a support ticket with Microsoft Azure to increase your quota limit for API call. For more information, see the following: https://docs.microsoft.com/en-us/azure/virtual-machines/troubleshooting/troubleshooting-throttling-errors
  
  Note - For testing purpose we have mentioned `throttlingIntervalSecs` field in default_eventhubbeat.yml , to set throttling interval in seconds (Negative values are not supported).

Authentication Error While Running EventHubBeat

• If you have deployed eventhubbeat on an Azure VM and receive any errors related to authentication failure or invalid resource name, check the authentication configuration in your Azure portal. After you fix the configuration, restart the eventhubbeat. The authentication information is described in the Event Hub Beat Using Azure Auth (MSI) topic.