Linux Troubleshooting Guide

Important File and Log Locations

Axon Agent v1.2.0 and Newer

File	Location	Purpose
Axon Agent Log	/var/logrhythm/logs/lragent/lragent.log	Log file for Axon Agent, containing logs related to Agent installation and running status.
Axon Agent Config File	/etc/logrhythm/lragent_config.json	Running configuration file for Axon Agent.
Axon Agent Binary	/bin/logrhythm/lr-agent	Binary for Axon Agent.
FluentD Log	/var/logrhythm/logs/fluent/fluentd-0.log	Log files for FluentD. Check here for error/info logs.
FluentD Supervisor Log	/var/logrhythm/logs/fluent/fluentd-supervisor-0.log	Additional log file for FluentD. Shows current running config, plugin versions and startup command.
FluentD Config	/etc/fluent/fluentd.conf	Config file for FluentD. Contains all settings for collectors, output, filtering and buffering. This file is automatically overwritten by Axon based on selections made in the UI. Manual changes will not be preserved if the Axon Agent service is running.
FluentD Binary	/opt/fluent/bin/fluentd	Binary for FluentD (td-agent).
FluentD Spool Folder	/opt/fluent/spool/axon/	Spool folder for incoming logs. This folder contains .txt and .meta file pairs of spooled logs waiting for processing. The maximum size of the spool folder is 64 GB. After that point, log loss will occur.

Axon Agent v1.1.8 and Older

File	Location	Purpose
Axon Agent Log	/var/log/logrhythm/lragent.log	Log file for Axon Agent, containing logs related to Agent installation and running status.
Axon Agent Config File	/etc/logrhythm/lragent_config.json	Running configuration file for Axon Agent.
Axon Agent Binary	/bin/logrhythm/lr-agent	Binary for Axon Agent.
FluentD Log	/var/log/td-agent/td-agent-0.log	Log files for FluentD. Check here for error/info logs.
FluentD Supervisor Log	/var/log/td-agent/td-agent-supervisor-0.log	Additional log file for FluentD. Shows current running config, plugin versions and startup command.
FluentD Config	/etc/td-agent/td-agent.conf	Config file for FluentD. Contains all settings for collectors, output, filtering, and buffering. This file is automatically overwritten by Axon based on selections made in the UI. Manual changes will not be preserved if the Axon Agent service is running.
FluentD Binary	/opt/td-agent/bin/fluentd	Binary for FluentD (td-agent).
FluentD Spool Folder	/opt/fluent/spool/axon/	Spool folder for incoming logs. This folder contains .txt and .meta file pairs of spooled logs waiting for processing. The maximum size of the spool folder is 64 GB. After that point, log loss will occur.

Diagnostic Script Usage for Linux

The Axon Agent Diagnostic script (lrDiagnosticsBash.sh) is included in your Axon Agent installation bundle. When creating a support case for an Axon Agent issue, include this diagnostic script with your submission.

To run the diagnostic script:

Run "lrDiagnosticsBash.sh" from the uncompressed .tar.gz installation bundle folder.
The script shows the running status while it collects information.
Allow the script to finish.
A .tar.gz named "lrAgentDiagnostic xxxxxx.tar.gz" is created in the same directory from which the lrDiagnosticsBash.sh script was executed.
The "xxxxxx" is the date/time stamp from when the information was collected.
Attach the lrAgentDiagnostic.tar.gz to your Axon Agent support case.

The diagnostic script covers all versions of Axon Agent, and seeing output in the console saying that certain files "cannot be found" is expected behavior. Not all file paths exist in all versions of Axon Agent.

Axon Agent General Troubleshooting Tips for Linux

Determining the Axon Agent's Running Status

To check if the service is running and has associated osquery and ruby processes, run the following command:

BASH

sudo systemctl status lr-agent.logrhythm

The following is an example of an output for a running Axon Agent:

Use the 'cat' command on the following log files and check for errors:

/etc/td-agent/td-agent.log
/etc/td-agent/td-agent.conf

Specific Errors and Resolutions

"Certificate has expired or is not yet valid" Error

The following error is showing up in your lragent.log file during installation of the LogRhythm Axon Agent:

BASH

"Fetching new config failed, falling back to old config...","msg":"transport error 
retrieving config: rpc error: code = Unavailable desc = connection error: desc = 
\"transport: authentication handshake failed: x509: certificate has expired or is 
not yet valid:

This error occurs when the system time on your Linux installation is incorrect. Update the current system date/time and this error will resolve.

After Installation, a "GetConfig" error is seen in the lragent log file (with debug logging turned on)

After installation, the following error is seen in the lragent log file:

CODE

{"level":"error","ts":"2024-03-20T12:39:00-06:00","msg":"Error in GetConfig","error: 
":"rpc error: code = Unauthenticated desc = ; retried get config with new node key: 
rpc error: code = Unauthenticated desc = "}
{"level":"error","ts":"2024-03-20T12:39:00-06:00","msg":"Error in GetConfig response","error: 
":"rpc error: code = Unauthenticated desc = "}
{"level":"info","ts":"2024-03-20T12:39:00-06:00","msg":"Node invalid","error:":"rpc error: 
code = Unauthenticated desc = "}
{"level":"debug","ts":"2024-03-20T12:39:00-06:00","msg":"Failed getting config due to 
permission error"}

This error is expected to be seen exactly once, before enrollment and immediately after installation. It does not indicate any problem with your Axon Agent installation. The first time the Axon Agent reaches out to Axon, it attempts to authenticate with an API key it does not yet have. After this error is seen, your Axon Agent will attempt to enroll and will be provided a valid API key.

A "no such host" error is seen in your lragent log file

The following error is caused by incorrectly set DNS settings on your operating system. If the Axon Agent cannot resolve the url indicated, it cannot communicate with Axon.

CODE

{"level":"info","ts":"2024-03-25T14:12:16-06:00","msg":"Node invalid","error:":"rpc 
error: code = Unavailable desc = connection error: desc = \"transport: Error while 
dialing: dial tcp: lookup app.na01.prod.boreas.cloud: no such host\""}

Try updating your DNS settings to a different primary DNS server.

"No such file or directory" error

You may see the following error during an agent installation:

CODE

cp: cannot stat '/some/filepath/lragent_config.json\n/some/other/filepath/lragent_config.json': no such file or directory

This is due to multiple copies of the lragent_config.json file existing on the machine. To resolve this, follow these steps:

Confirm there is no agent currently installed on the machine.
Use the following command to ensure there is only one copy of the file listed, in the directory where the Agent installers exist:
CODE
```
find / -type f -name lragent_config.json
```

Once extra copies of the config file have been removed from the machine, the installation will run as expected.