This page provides guidance for applying security patches to both Windows and CentOS-based operating systems used for the LogRhythm SIEM, NetMon, and Open Collector. Each section describes a way to patch for online (internet-connected) and offline systems.

With the exception of NetMon appliances, LogRhythm recommends that the latest security patches from the OS vendor be applied. Users should patch the underlying OSs as part of their regular patching cycle per their internal patching policy.

Microsoft Windows

To run online Windows Updates from the Microsoft repositories, follow the steps in this Microsoft article: https://support.managed.com/kb/a2071/how-to-install-windows-updates-on-a-windows-server.aspx

To download updates from the Windows catalog and distribute them manually, follow the steps in this Microsoft article: https://support.microsoft.com/en-gb/help/323166/how-to-download-updates-that-include-drivers-and-hotfixes-from-the-win

Patch Microsoft SQL (XM or PM)

The latest SQL service packs are fully tested and supported by LogRhythm. For the latest updates per SQL server version, see https://docs.microsoft.com/en-us/sql/database-engine/install-windows/latest-updates-for-microsoft-sql-server?view=sql-server-ver15.

To update a DR or HA deployment, see Patch Disaster Recovery-Enabled Deployments or Patch High Availability-Enabled Deployments.

Patch Disaster Recovery-Enabled Deployments (XM or PM)

Windows Operating System

Patch the Windows Server OS following your regular internal patching policy.

SQL Server

This procedure assumes that the DR failover process has been successfully tested in your environment. If this is not the case, remediate the DR failover process, and then proceed with the steps below.

DR-enabled SQL Servers should not be subject to automatic updates (such as Service Packs and Cumulative Updates) on the SQL Server instance. To avoid database corruption, these updates should be applied in a specific pattern. It is highly recommended to exclude DR-enabled LogRhythm systems from automatic patches to the SQL Server, though OS updates are still advised.

To patch SQL Server 2016:

  1. Perform the service pack or cumulative update on the secondary replica.

  2. Use the DR Control utility to perform a cluster failover of all database availability groups from the primary replica to the secondary replica.

  3. Perform the service pack or cumulative update on the primary replica.

  4. Use the DR Control utility to perform a cluster failback of all database availability groups from the secondary replica to the primary replica.

    It is recommended to perform a system reboot immediately after performing the service pack or cumulative update on the SQL Server instance, even if the installer does not call for one.

For further detail on SQL patching Always On Availability Groups, see https://docs.microsoft.com/en-us/sql/database-engine/availability-groups/windows/upgrading-always-on-availability-group-replica-instances?view=sql-server-2016.

Patch High Availability-Enabled Deployments (XM or PM)

Windows Operating System

Patch the Windows Server OS following your regular internal patching policy.

SQL Server

The LogRhythm HA solution employs full-disk mirroring between protected nodes to synchronize the drives that contain the PM SQL databases (both system and LogRhythm). The system databases reside on the mirrored volume for the D: drive. When a SQL Server service pack or cumulative update is applied to an instance of SQL Server, the system (and LogRhythm) databases are also updated with the build number. When the master database version is modified with the build number from the SQL Server update, that change is mirrored to the inactive nodes via DataKeeper. If the inactive node were brought online in the middle of the patching process, the SQL Server executable would recognize that the master database build number is not consistent with the version that the service executable expects. This would cause the database engine to fail to start.

It is strongly recommended to exclude all HA nodes from automatic SQL Server service packs and cumulative updates. This recommendation is applicable to SQL Server updates only—Windows security patches should still be applied per internal security policy.

On the currently inactive HA node:

  1. Open DataKeeper.

  2. Ensure that all mirror jobs are in a status of mirroring & synchronized.

  3. Right-click each mirror job, and then click Pause and unlock All Mirrors.

  4. Start SQL Server services on the inactive HA node.

  5. Install the SQL Server updates.

  6. If a reboot is prompted:

    Reboot is recommended.
    1. Reboot the node, and then log in again.

    2. Open DataKeeper.

    3. Right-click each mirror job, and then click Pause and unlock All Mirrors.
    4. Start SQL Server services on the inactive HA node.
    5. Proceed to step 8.

  7. If no reboot is prompted, proceed to step 8.

  8. Confirm successful SQL Server startup via the ERRORLOG and login availability via SQL Server Management Studio.

  9. Manually stop the SQL Server service on the node.

  10. (Optional) Repeat these steps for any additional inactive HA nodes.

On the currently active HA node:

  1. Open DataKeeper.

  2. Confirm that all mirrors are in a status of paused & unlocked. If necessary, right-click each mirror job, and then click Pause and Unlock All Mirrors.

  3. Click the LifeKeeper GUI, and then set the top-level ResTag to Out of Service.

  4. Manually stop all LogRhythm Services on the node.

    Leave the SQL Server services running. They are required for the SQL Server update.
    After this step, LogRhythm services will be non-functional.
  5. Install the SQL Server updates.

  6. If reboot is prompted:

    Reboot is recommended.
    1. Reboot the node, and then log in again.

    2. Wait for SQL and all LogRhythm services to start. This could take 10-15 minutes.

    3. Proceed to step 8.

  7. If no reboot is prompted, proceed to step 8.

  8. Confirm successful SQL Server startup via the ERRORLOG and login availability via SQL Server Management Studio.

  9. Confirm that all LogRhythm services are started and operational by checking the various component logs.

  10. Open DataKeeper on the node.

  11. Right-click each mirror job, and then click Continue and Lock All Mirrors.

  12. Click the LifeKeeper GUI, and then set the top-level ResTag to In Service.

  13. Wait for the disk mirror jobs to synchronize fully.

  14. In DataKeeper, verify that all mirror jobs are fully synchronized.

  15. (Optional) Perform a full switchover of resources to the secondary node to verify HA resource protection, then perform a full switchback of resources to the primary node to verify HA resource protection.

CentOS-Based Data Indexer

  • LogRhythm DX servers must be patched with CentOS base repositories only. Other repositories should not be created or enabled on DX servers. Other packages, including Elasticsearch, must only be updated as part of a LogRhythm upgrade. If you detect any vulnerabilities after following these instructions, please contact LogRhythm Support.
  • Currently, LogRhythm only supports CentOS version 7. DXs should not be upgraded to CentOS version 8.
  • Before patching using CentOS repositories, LogRhythm 7.1.x must be upgraded to 7.2.x or later.

Because of the lack of necessary metadata in the yum repositories, "yum-plugin-security" is non-functional on CentOS. Alternatively, yum update updates all packages, including security patches.

Update Online Systems

To update online systems with CentOS repositories, update DX servers against the CentOS official base repository. Additional repositories should not be used for DX patching.

yum --disablerepo=* --enablerepo=base update
BASH

Update Offline Systems

To update offline systems, choose one of these three methods:

For a local yum server with locally mirrored repositories, the OS for LogRhythm DX appliances can also be updated. If DX appliances do not have internet access, LogRhythm recommends creating a local repository to update DX servers.

There are multiple ways to install and configure a local repository for CentOS Linux. The following instructions are provided as an example.

Use the CentOS Official Image File

Updating CentOS using an .iso file is a fast way to patch DX servers, as it does not require creating a local repository.

Although CentOS releases new image files frequently, compared to CentOS online repositories, these images may not contain the most recent version of packages, as CentOS releases a new ISO every eight weeks.
  1. Download the latest official CentOS 7.x minimal .iso file. The mirror links for CentOS images can be found at http://isoredirect.centos.org/centos/7/isos/x86_64/.
  2. Copy the .iso file to the DX appliance.
  3. If it does not already exist, create the directory /media/CentOS:

    mkdir -p /media/CentOS
    BASH
  4. Use the following command to mount the .iso locally, replacing "$FilePath.iso" to full path of the CentOS .iso file:

    mount -t iso9660 -o loop $FilePath.iso /media/CentOS/
    BASH

    For example, if the downloaded .iso filename is "loopCentOS-7-x86_64-Minimal-2003.iso" and it is copied to /home/logrhythm/, the mount command would be:

    mount -t iso9660 -o loop /home/logrhythm/loopCentOS-7-x86_64-Minimal-2003.iso /media/CentOS/
    BASH
  5. Create or edit a local yum repository file named "/etc/yum.repos.d/CentOS-Media.repo" and update the file address to point to the directory mounting the .iso file:

    # CentOS-Media.repo
    #
    #  This repo can be used with mounted DVD media, verify the mount point for
    #  CentOS-7.  You can use this repo and yum to install items directly off the
    #  DVD ISO that we release.
    #
    # To use this repo, put in your DVD and use it with the other repos too:
    #  yum --enablerepo=c7-media [command]
    #
    # or for ONLY the media repo, do this:
    #
    #  yum --disablerepo=\* --enablerepo=c7-media [command]
    
    [c7-media]
    name=CentOS-$releasever - Media
    baseurl=file:///media/CentOS/
            file:///media/cdrom/
            file:///media/cdrecorder/
    gpgcheck=1
    enabled=1
    gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
    
    
    BASH
  6. Run the yum update command:

    sudo yum clean all 
    yum --disablerepo=* --enablerepo=c7-media update
    
    
    BASH
  7. When the update is complete, disable the repository by editing the file "/etc/yum.repos.d/CentOS-Media.repo" and changing the enabled value to 0:

    # CentOS-Media.repo
    #
    #  This repo can be used with mounted DVD media, verify the mount point for
    #  CentOS-7.  You can use this repo and yum to install items directly off the
    #  DVD ISO that we release.
    #
    # To use this repo, put in your DVD and use it with the other repos too:
    #  yum --enablerepo=c7-media [command]
    #
    # or for ONLY the media repo, do this:
    #
    #  yum --disablerepo=\* --enablerepo=c7-media [command]
    
    [c7-media]
    name=CentOS-$releasever - Media
    baseurl=file:///media/CentOS/
            file:///media/cdrom/
            file:///media/cdrecorder/
    gpgcheck=1
    enabled=0
    gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
    
    BASH
  8. Unmount the .iso file:

    umount /media/CentOS/
    BASH

Create a Local Repository and Update the DX Servers Using an HTTP Server

  1. Install a new CentOS 7.x server with internet access. To verify whether the correct version of CentOS is installed, run the following command on the offline repository server, then run the same command on DX servers and confirm that the values for "arch," "releasever," "infra," and "basearch" are identical.

    sudo python -c 'import yum, json; yb = yum.YumBase(); print json.dumps(yb.conf.yumvar,indent=2)'
    BASH
  2. Enable an EPEL repository on the server to install nginx packages:

    sudo yum install epel-release 
    BASH
  3. On the repository server, install the "createrepo yum-utils" and "nginx" packages:

    sudo yum install createrepo yum-utils nginx 
    BASH
  4. Set the OS firewall to allow HTTP.

    As part of the installation, the HTTP port should be allowed by default.
    sudo firewall-cmd --zone=public --permanent --add-service=http
    BASH
  5. Create a directory named "repository":

    sudo mkdir -p /var/www/repository
    BASH
  6. Run the following four commands separately to download and update the local repository for the base, centosplus, extras, and updates repositories. To download the most recent updates from the CentOS repository prior to any subsequent patching run, repeat these commands.

    sudo reposync -g -l -d -m --repoid=base --newest-only --download-metadata --download_path=/var/www/repository/
    sudo reposync -g -l -d -m --repoid=centosplus --newest-only --download-metadata --download_path=/var/www/repository/
    sudo reposync -g -l -d -m --repoid=extras --newest-only --download-metadata --download_path=/var/www/repository/
    sudo reposync -g -l -d -m --repoid=updates --newest-only --download-metadata --download_path=/var/www/repository/
    BASH
  7. When all packages are synchronized, use the createrepo command to create or update the repodata repository. Run the command each time the reposync command is used.

    sudo createrepo /var/www/lighttpd/repository/ 
    BASH
  8. Rename the /etc/nginx/nginx.conf file to /etc/nginx/nginx.conf.oem:

    sudo mv /etc/nginx/nginx.conf /etc/nginx/nginx.conf.oem 
    BASH
  9. Create a new /etc/nginx/nginx.conf file:

    sudo vi /etc/nginx/nginx.conf
    BASH
  10. Add the following content:

    user nginx;
    worker_processes auto;
    error_log /var/log/nginx/error.log;
    pid /run/nginx.pid;
    
    # Load dynamic modules. See /usr/share/doc/nginx/README.dynamic.
    include /usr/share/nginx/modules/*.conf;
    
    events {
        worker_connections 1024;
    }
    
    http {
        log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                          '$status $body_bytes_sent "$http_referer" '
                          '"$http_user_agent" "$http_x_forwarded_for"';
    
        access_log  /var/log/nginx/access.log  main;
    
        sendfile            on;
        tcp_nopush          on;
        tcp_nodelay         on;
        keepalive_timeout   65;
        types_hash_max_size 2048;
    
        include             /etc/nginx/mime.types;
        default_type        application/octet-stream;
    
        # Load modular configuration files from the /etc/nginx/conf.d directory.
        # See http://nginx.org/en/docs/ngx_core_module.html#include
        # for more information.
        include /etc/nginx/conf.d/*.conf;
    
     server {
            listen       80 default_server;
            listen       [::]:80 default_server;
            server_name  _;
            root         /var/www/repository;
    
            # Load configuration files for the default server block.
            include /etc/nginx/default.d/*.conf;
    
            location / {
                    allow all;
                    sendfile on;
                    sendfile_max_chunk 1m;
                    autoindex on;
                    autoindex_exact_size off;
                    autoindex_format html;
                    autoindex_localtime on;
            }
              error_page 404 /404.html;
                location = /40x.html {
            }
    
              error_page 500 502 503 504 /50x.html;
                location = /50x.html {
            }
    
        }
    }
    BASH
  11. On the DX servers, create a new yum repository file:

     sudo vi /etc/yum.repos.d/CentOS-Local.repo
    BASH
  12. Copy the following content into the CentOS-Local.repo file, replacing $IP in front of "baseurl" with the IP address of the offline repository server:

    [LocalRepository]
    name=LocalRepository
    baseurl=http://$IP/
    gpgcheck=1
    gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
    enabled=1
    BASH
  13. Run the yum update command on the DX server:

    sudo yum clean all 
    sudo yum update 
    BASH

Copy Repository Files to DX Servers and Run the Update Command Against the Repository on the Server's Local Storage

Instead of using the local network to access to the repository server, a repository can be created and copied to each DX server manually. They are then updated with the local repository available on the local disk. 

  1. Install a new CentOS 7.x server with internet access. To verify whether the correct version of CentOS is installed, run the following command on the offline repository server, and then run the same command on DX servers to confirm that the values for "arch," "releasever," "infra," and "basearch" are identical.

    sudo python -c 'import yum, json; yb = yum.YumBase(); print json.dumps(yb.conf.yumvar,indent=2)'
    BASH
  2. On the repository server, install the following packages:

    sudo yum install createrepo yum-utils 
    BASH
  3. Create a directory named repository on volume "var":

    sudo mkdir -p /var/repository
    BASH
  4. Run the four following commands separately to download and update the local repository for the base, centosplus, extras, and updates repositories. To download the most recent updates from the CentOS repository prior to any subsequent patching run, repeat these commands.

    sudo reposync -g -l -d -m --repoid=base --newest-only --download-metadata --download_path=/var/repository/
    sudo reposync -g -l -d -m --repoid=centosplus --newest-only --download-metadata --download_path=/var/repository/
    sudo reposync -g -l -d -m --repoid=extras --newest-only --download-metadata --download_path=/var/repository/
    sudo reposync -g -l -d -m --repoid=updates --newest-only --download-metadata --download_path=/var/repository/
    BASH
  5. When all packages sync, use the createrepo command to create and update the repodata repository. Run the following command each time reposync commands are re-run for subsequent patching.

    sudo createrepo /var/repository/
    BASH
  6. Create a tar.gz file from the repository directory. For example, create the tar file in the /home/logrhythm/ directory.

    sudo tar czf /home/logrhythm/repository.tar.gz  /var/repository/ 
    BASH
  7. Copy repository.tar.gz to the /home/logrhythm/ directory on the DX server, and then extract the tar file using the following command:

    The volume "/var/" has limited space. If the repository size is larger than the available space on the volume, extract the tar file to a different location.
    sudo tar xzf /home/logrhythm/repository.tar.gz -C /
    BASH

    The repository files should be extracted to /var/repository/ on the DX server.

  8. On the DX servers, create a new yum repository file:

     sudo vi /etc/yum.repos.d/CentOS-Local.repo
    BASH
  9. Copy the following content to the file in CentOS-Local.repo, where the value /var/repository/ in front of "baseurl" is the location to which repository.tar.gz was extracted.

    [LocalRepository]
    name=LocalRepository
    baseurl=file:///var/repository
    gpgcheck=1
    gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
    enabled=1
    BASH
  10. Run the yum update command on the DX server:

    sudo yum clean all 
    sudo yum update 
    BASH
  11. When the yum update is complete, edit the repo file /etc/yum.repos.d/CentOS-Local.repo again:

     sudo vi /etc/yum.repos.d/CentOS-Local.repo
    BASH
  12. To disable the repository, set the value of Enabled to 0:

    [LocalRepository]
    name=LocalRepository
    baseurl=file:///var/repository
    gpgcheck=1
    gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
    enabled=0
    BASH
  13. After the update is complete, remove the repository files.

Restart DX Servers

The Linux kernel is only loaded into memory during the boot process. To verify whether the new kernel is installed on the OS, check the version of the running kernel and compare it to the list of of installed versions.

Check If an OS Reboot Is Required

  1. To show the version of the current kernel OS, run the following command:

    sudo uname -r 
    BASH
  2. To show kernel versions installed on the system, run the following command:

    sudo rpm -qa kernel
    BASH
  3. If there is a newer kernel version available in the rpm -qa output, restart the OS to load the new kernel.

    Restarting a single node DX introduces downtime for DX, and it can take a significant amount of time to recover and resume indexing.

Restart a Single-Node Cluster

  1. Stop indexing and perform a synced flush to speed up shard recovery:

    curl -XPOST "localhost:9200/_flush/synced?pretty"
    BASH
  2. Check the response to make sure there are no failures. Synced flush operations that fail due to pending indexing operations are listed in the response body, although the request itself still returns a 200 OK status. If there are failures, reissue the request.

  3. Shut down all DX services on the server:

    sudo /usr/local/logrhythm/tools/stop-all-services-linux.sh
    BASH
  4. Restart the DX server:

    Before restarting the servers, verify iDRAC access to the server.
     sudo systemctl --force reboot
    BASH
  5. When the node restarts, verify that the cluster is stable (green). This can take a significant amount of time. To monitor the cluster, use the following command:

     watch 'curl -s "localhost:9200/_cluster/health?pretty"'
    BASH

Rolling Restart a Multi-Node Cluster

The following steps should be run on one node at a time. For example, if there are three nodes in a DX cluster, run these steps on each node sequentially.

  1. Disable shard allocation:

    curl -XPUT localhost:9200/_cluster/settings?pretty -H 'Content-Type: application/json' -d '{"transient":{"cluster.routing.allocation.enable":"none"}}'
    BASH
  2. Stop indexing and perform a synced flush to speed up shard recovery:

    curl -XPOST "localhost:9200/_flush/synced?pretty"
    BASH
  3. Check the response to make sure there are no failures. Synced flush operations that fail due to pending indexing operations are listed in the response body, although the request itself still returns a 200 OK status. If there are failures, reissue the request.

  4. Stop all DX services on one server at a time:

    sudo /usr/local/logrhythm/tools/stop-all-services-linux.sh
    BASH
  5. Restart the node:

    Before restarting the node, verify iDRAC access to the server.
     sudo systemctl --force reboot
    BASH
  6. Confirm the node you restarted joins the cluster by checking the log file or by submitting a _cat/nodes request on any other node in the cluster:

     watch 'curl -s  -XGET "localhost:9200/_cat/nodes?pretty&s=name&v"'
    BASH
  7. After the node has joined the cluster, re-enable shard allocation. To enable shard allocation and start using the node, remove the cluster.routing.allocation.enable setting:

    curl -X PUT "localhost:9200/_cluster/settings?pretty" -H 'Content-Type: application/json' -d'{"transient": { "cluster.routing.allocation.enable": "all" }}'
    BASH
  8. (Optional) To speed up the recovery process, increase the limit of total inbound and outbound recovery traffic and concurrent incoming shard recoveries allowed on a node:

    curl -XPUT localhost:9200/_cluster/settings?pretty -H 'Content-Type: application/json' -d'{"transient":{"cluster.routing.allocation.node_concurrent_recoveries": 10}}'
    curl -XPUT localhost:9200/_cluster/settings?pretty -H 'Content-Type: application/json' -d '{"transient":{"indices.recovery.max_bytes_per_sec": "1000mb"}}'
    BASH
  9. When the node has recovered and the cluster is green, repeat these steps for each node that needs to be restarted.

  10. When the node has restarted, verify the cluster until it is stable (green). To monitor the cluster, use the following command:

     watch 'curl -s "localhost:9200/_cluster/health?pretty"'
    BASH

LogRhythm NetMon

Patches are applied with NetMon version updates. It is recommended to update the NetMon to its latest version. If additional packages need to be updated, raise a ticket with LogRhythm Support to add the package into next available NetMon release.

Yum Update

Running a yum update on a NetMon system is not supported.

CentOS-Based LogRhythm Open Collector

To update an online system with CentOS repositories, run the following command:

yum clean all
sudo yum update 
BASH

To update an offline system with local repositories, see the following reference materials:

Update Firmware and Drivers on Dell EMC PowerEdge Servers

To access the most recent firmware from Dell's website, contact Dell Support at this link (Dell service tag required): https://www.dell.com/support/home/en-us?app=products

To run firmware and drivers updates for Dell servers, see https://www.dell.com/support/article/en-us/sln300662/updating-firmware-and-drivers-on-dell-emc-poweredge-servers#WhatarethedifferentsmethodstoupdateaPowerEdge.

For information on iDRAC and BIOS using the IDRAC interface, see https://www.dell.com/support/article/en-us/sln292363/dell-poweredge-update-the-firmware-of-single-system-components-remotely-using-the-idrac?lang=en.