Windows Troubleshooting Guide

Important File and Log Locations

Axon Agent v1.2.0 and Newer

File	Location	Purpose
Axon Agent Log	C:\var\logrhythm\logs\lragent\lragent.log	Log file for Axon Agent, containing logs related to Agent installation and running status.
FluentD Log	C:\var\logrhythm\logs\fluent\fluentd-0.log	Log file for FluentD. Check here for error/info logs.
FluentD Supervisor Log	C:\var\logrhythm\logs\fluent\fluentd-supervisor-0.log	Additional log file for FluentD. Shows current running config, plugin versions and startup command.
FluentD Config	C:\opt\fluent\etc\fluent\fluentd.conf	Config file for FluentD. Contains all settings for collectors, output, filtering, and buffering. This file is automatically overwritten by Axon based on selections made in the UI. Manual changes will not be preserved if the Axon Agent service is running.
FluentD Spool Folder	C:\opt\fluent\spool\axon	Spool folder for incoming logs. This folder contains .txt and .meta file pairs of spooled logs waiting for processing. The maximum size of the spool folder is 64 GB. After that point, log loss will occur.
osQuery Logs	C:\var\logrhythm\logs\osquery	Log Files for osQuery.

Axon Agent v1.1.8 and Older

File	Location	Purpose
Axon Agent Log	C:\var\logrhythm\lragent.log	Log file for Axon Agent, containing logs related to Agent installation and running status.
FluentD Log	C:\opt\td-agent\td-agent-0.log	Log files for FluentD. Check here for error/info logs.
FluentD Supervisor Log	C:\opt\td-agent\td-agent-supervisor-0.log	Additional log file for FluentD. Shows current running config, plugin versions, and startup command.
FluentD Config	C:\opt\td-agent\etc\td-agent\td-agent.conf	Config file for FluentD. Contains all settings for collectors, output, filtering, and buffering. This file is automatically overwritten by Axon based on selections made in the UI. Manual changes will not be preserved if the Axon Agent service is running.
FluentD Spool Folder	C:\opt\td-agent\spool\axon	Spool folder for incoming logs. This folder contains .txt and .meta file pairs of spooled logs waiting for processing. The maximum size of the spool folder is 64 GB. After that point, log loss will occur.

Diagnostic Script Usage for Windows

The Axon Agent Diagnostic script (lrDiagnostics.ps1) is included in your Axon Agent installation bundle. When creating a support case for an Axon Agent issue, include this diagnostic script with your submission.

The Windows diagnostic script may not be compatible with PowerShell versions prior to version 5. To check your PowerShell version, run the following command in PowerShell: $PSVersionTable

To run the diagnostic script:

Double-click "lrDiagnostics.ps1" from the unzipped installation bundle folder.
A PowerShell window opens and shows the status of the script.
Allow the script to finish and do not close the PowerShell window.
A .zip file named "lrAgentDiagnostics xxxxxx.zip" is created in the same directory from which the lrDiagnostics.ps1 script was executed.
The "xxxxxx" is the date/time stamp from when the information was collected.
Attach the lrAgentDiagnostics.zip to your Axon Agent support case.

The diagnostic script covers all versions of Axon Agent, and seeing output in the console saying that certain files "cannot be found" is expected behavior. Not all file paths exist in all versions of Axon Agent.

Axon Agent General Troubleshooting Tips for Windows

Confirm Proper Connectivity to the Axon Signal Ingest Service

Run the following command to test a system's connectivity to the Axon backend:

CODE

Invoke-RestMethod -Uri https://api.na01.prod.boreas.cloud/signal-ingest-svc/openapi

A successful test prints a large block of text to the CLI, with the first few lines looking like this:

CODE

openapi: 3.0.3
info:
title: SignalIngest Service
description: SignalIngest Service
version: 20240418222913-e026ffa

Determine the Axon Agent's Running Status

Check the following locations to determine the Axon Agent's status:

Check the Windows services.msc page for the "LogRhythm Axon Agent" service. It should be in "Running" status, with a Startup Type value of "Automatic (Delayed start)".
Check Windows task manager; you should see 2 "Ruby" processes running if the Agent was successfully able to start FluentD.
Tail the FluentD Log, FluentD Supervisor Log, and Axon Agent Log from the Important Log and File Locations table. Check these logs for errors.

Not Collecting Any Logs

In the event that no logs are being collected, check the following items:

Check that the time on your Agent machine matches the time you are searching in the UI.

It is recommended to widen the search time window in the past and future.
Check to make sure LogrhythmAgentSvc is running in the machine’s Services window:
Run the following command in PowerShell to check for the required running pipes:
1. Prior to Windows 10:
  1. [System.IO.Directory]::GetFiles("\\.\\pipe\\")
2. All other versions of Windows:
  1. ```
  get-childitem \\.\pipe\
```
3. Check for the following output:
  2. If these pipes are not seen while the LogRhythm Axon Agent service is running, open a Customer Service Support Case
Navigate to C:\opt\logrhythm\conf and open the json file. Verify that the sipUrl field is correct:
Navigate to C:\opt\logrhythm\conf and verify that the database (db) file exists, and that the size is 64 KB.

If the file exists but is only 32 KB, allow a minute or so to pass.
If the file is still 32 KB after a minute, the enrollment key for this agent is broken (see the Authorization Problems section below).
Navigate to C:\opt\logrhythm\conf and verify that pidfile and fluent.pidfile are present.
If so, verify that the IDs in those files match the running process's PID in the task manager details.

Task Manager should show two instances of exe running, two instances of ruby.exe running for FluentD, and one instance of lr-agent.exe running
Navigate to C:\opt\td-agent\etc\td-agent and verify that the plugins folder exists, and that it contains an RB file.
Navigate to C:\opt\td-agent\etc\td-agent and verify that td-agent.conf exists:

If the file doesn’t exist, restart the LogrhythmAgentSvc and wait five minutes. If the file still doesn’t exist, the configuration for the agent is broken (see the Authorization Problems section below).
Navigate to C:\var\logrhythm and open the latest Agent log:

If the last log message is Enrolling… but there is no subsequent log message detailing the Agent ID, the enrollment key for this agent is broken (see the Authorization Problems section below).
Verify that there are no errors in the log since the last LogrhythmAgentSvc startup?
If the log isn’t helpful, follow the steps below:
1. In the Agents page of the Axon UI, click on the Agent you are using to see the Agent details.
2. Use the Edit Agent button to set the Log Level to "Debug."
3. Navigate back to C:\var\logrhythm and open the latest Agent log.
4. At the debug level, you are looking for several key messages (if these aren’t all appearing, see the Authorization Problems section below):
  1. “Configuration: <json payload>”
  2. “Writing to file”
  3. “Restarting fluentd”

Expected Logs Not Being Collected

If the logs you're expecting aren't being collected, do the following:

In the Axon UI, verify that the Agent Profile you used to generate this bundle has the collectors you expect.
Check that the time on your Agent machine matches the time you are searching in the UI.

It is recommended to widen the search time in the past and future.
Navigate to C:\opt\td-agent\etc\td-agent and open td-agent.conf.
1. Verify that the items in here match what you expect based on your profile.
  
  The conf file has the collector IDs associated with this Agent (for example, Syslog).
2. Verify that the collectors in the UI match as expected.
  1. If there is a syslog collector, verify that the Windows Defender Firewall (in advanced settings) is allowing inbound syslog traffic.
    You can look for the named rule you created when setting it up, or sort by port number to find the port you are using.
  2. Verify that the machine flat files exist as outlined in C:\opt\td-agent\etc\td-agent\td-agent.conf
If only the Syslog collector logs aren't showing up in the Axon UI:
1. Change the Syslog collector Log Level to "Debug" in the Edit Agent Profile form.
2. Open the C:\opt\td-agent\td-agent-0.log file.
3. If there is a log containing this message, this means that you have more than one listener on the configured port. Windows only allows one listener per port. This is likely to happen if the LR7 Sysmon Agent is also configured on the machine to collect syslogs.
  CODE
```
unexpected error error_class=Errno::EADDRINUSE error="Only one usage of each socket address (protocol/network address/port) is normally permitted. - bind(2)"
```
4. Complete one of these solutions:
  1. Turn off other listener.
  2. Configure collection on a different port (requires configuring your devices producing logs to send to the new port, as well as changing the port number in the Edit Agent Profile form in the Syslog collector details).

Authorization Problems

If you are encountering authorization problems, try the following steps:

Verify that the agent can actually reach the provided URL, and ensure that the Windows Defender Firewall (or other firewall software) is open for outbound connections on port 443.
The Agent never accepts inbound connections, only outbound.
Navigate to C:\opt\logrhythm\conf\lragent_config.json and confirm that the tenant ID is correct.
Subsequent steps to diagnose authorization problems rely on non-UI actions against the LogRhythm Axon API.

In this event, it is recommended to contact LogRhythm Customer Support.

Generate an Axon Agent .msi installation log file

To generate a log file for the Axon Agent .msi installation, use the following command:

CODE

msiexec /i "C:\<your filepath here>\Axon_Agent_Installer_x.x.x.msi" /l*v "C:\<your filepath here>\Axon_install.log"

The first file path is the path to the .msi installer for Axon Agent, and the second file path is the path to where the log file will be written.

Specific Errors and Resolutions

After Uninstalling the Windows Axon Agent, the C:\Opt and C:\Var folders are Still Present

During uninstallation, if the LogRhythm Axon Agent service is left running, the opt and var folders will not be deleted. If the agent service is stopped before uninstall, those folders are properly deleted.

If the Axon Agent has not yet been uninstalled, do the following:

Manually stop the "LogRhythm Axon Agent" service in services.msc.
Uninstall the Axon Agent as usual through the Add or Remove Programs menu.
The Opt and Var folders will now be properly deleted during uninstallation.
Reboot your machine.

If the Axon Agent has already been uninstalled and the opt and var folders remain on the C:\ drive, manually delete the Opt and Var folders from the C:\ drive.

The LogRhythm Axon Agent Service Does Not Stop When Manually Stopped Through services.msc

If the Agent service is stopped during active enrollment, it may hang or fail to cleanly stop.

Give the Axon Agent 5-10 minutes post-install to enroll and download a configuration before stopping the service.

After Uninstalling the Windows Axon Agent, the LogRhythm Axon Agent Service is Still Visible in the services.msc List

After uninstallation, the LogRhythm Axon Agent service may not be removed from the Services list, but instead moved to “disabled” status.

Rebooting the operating system removes the LogRhythm Axon Agent service from the list.

After Installation, a "GetConfig" Error is seen in the lragent log file (with debug logging turned on)

After installation, the following error appears in the lragent log file:

CODE

{"level":"error","ts":"2024-03-20T12:39:00-06:00","msg":"Error in GetConfig",
"error: ":"rpc error: code = Unauthenticated desc = ; retried get config with 
new node key: rpc error: code = Unauthenticated desc = "}
{"level":"error","ts":"2024-03-20T12:39:00-06:00","msg":"Error in GetConfig response","error: 
":"rpc error: code = Unauthenticated desc = "}
{"level":"info","ts":"2024-03-20T12:39:00-06:00","msg":"Node invalid","error:":"rpc error: 
code = Unauthenticated desc = "}
{"level":"debug","ts":"2024-03-20T12:39:00-06:00","msg":"Failed getting config due to 
permission error"}

This error is expected to be seen exactly once, before enrollment and immediately after installation. It does not indicate any problem with your Axon Agent installation. The first time the Axon Agent reaches out to Axon, it attempts to authenticate with an API key it does not yet have. After this error is seen, your Axon Agent will attempt to enroll and will be provided a valid API key.

A "no such host" error is seen in your lragent log file

The following error is caused by incorrectly set DNS settings on your operating system. If the Axon Agent cannot resolve the url indicated, it cannot communicate with Axon.

CODE

{"level":"info","ts":"2024-03-25T14:12:16-06:00","msg":"Node invalid","error:":"rpc 
error: code = Unavailable desc = connection error: desc = \"transport: Error while 
dialing: dial tcp: lookup app.na01.prod.boreas.cloud: no such host\""}

Try updating your DNS settings to a different primary DNS server.

The following terminal command will tell you if your DNS settings are working:

CODE

nslookup app.na01.prod.boreas.cloud