HA Upgrade Procedure Checklist

This document provides a checklist to reference when upgrading your LogRhythm High Availability (HA) deployment.

This checklist is not intended to replace the LogRhythm HA Upgrade Guide, which contains important information about the upgrade process. It is strongly recommended that you perform your upgrade by following the detailed procedures in that guide, using this checklist to track your progress.

This checklist assumes that your deployment is not a standard deployment or in DR. For those deployments, use the standard upgrade or DR checklist.

Prepare for the Upgrade

• Get the latest version of the LogRhythm HA Upgrade Guide from docs.logrhythm.com (or download the PDF).
• Review the upgrade requirements and considerations.
  
  • HA Upgrades
    

    • Cloud infrastructure is not supported in HA environments.
    • As part of the upgrade, SteelEye DataKeeper version will be upgraded to 8.7.0 automatically.
  • Scheduling the upgrade
  • FIPS Mode support
  • Core services and Client Console compatibility
  • SQL Server security hardening impacts
  • System Monitor Agent considerations
  • OS requirements for the Windows components
  • Microsoft .Net Framework 4.7.2 requirements
  • Web Console prerequisites
  • Component compatibility
• Ensure that you have the required Administrator credentials.
  • Local Administrator privileges for the LogRhythm servers
  • SQL Server password for the LogRhythmAdmin account
  • SQL Server sa password for the LogRhythm Databases

  • LogRhythm Service accounts referenced within the Configuration Manager

    

    A list of these passwords can be found in the LogRhythm Default Passwords section of docs.logrhythm.com.
• Download software to upgrade a LogRhythm deployment.
  • HA Upgrade 7.x.zip file
  • LogRhythm Database Upgrade Tool
  • LogRhythm Install Wizard
  • Linux Data Indexer Installer
    • Required only if you have a Linux Data Indexer
  • (Optional) System Monitor Packages
  • (Optional) Threat Intelligence Service
  • (Optional) TrueIdentity Sync Client
  • (Optional) SOAP API
• Record service credentials.
  • LifeKeeper
  • SIOS DataKeeper

• Request a LogRhythm license file at least one business day prior to upgrade.

  

  If you are upgrading to a new LogRhythm major version, a license file is required.
• Modify web.config for the LR API.
  • Required only if you are using the LR API
• Note Web Console environmental variables.
  • Required only if you are overriding the Configuration Manager settings on one or more Web Console servers
    
• Note Platform Manager IP, LogRhythm Web UI password, and login warning banner.
• Synchronize stored Knowledge Base
  • Required only if you have downloaded a Knowledge Base but have not yet synchronized it
• Configure the System Monitor service.
• Verify deployment status in the LogRhythm Infrastructure Installer.
• Within the LifeKeeper GUI, shut down the LogRhythm Solution. When all services are shut down:
  • Place the VIP back In Service.
  • Start the LogRhythm Service Registry on both the primary and secondary HA servers.
• Shut down antivirus and endpoint protection software.

• Exit all LogRhythm Client Consoles.

Upgrade the LogRhythm Deployment

• Stop the LogRhythm core services on Windows appliances.
  • Platform Manager Servers
  • • Alarming and Response Manager
    • Job Manager
    • AI Engine Cache Drilldown
  • Data Processor Servers
    
    • Mediator Server Service
  • AI Engine Servers
    
    • AI Engine
    • AI Engine Communication Manager
  • Web Console Servers
    
    • Web Services Host API
    • Web Indexer
    • Web Console UI
    • Web Console API
    • Case API
  • (Optional) Kibana
  

  • Exit all LogRhythm Client Consoles.
  • System Monitor Agents can remain running throughout the upgrade.
  • Any services not mentioned in the above list are not required to be stopped for the upgrade. The Database Upgrade Tool might stop a few additional services on the PM/XM, but this is expected behavior.

• Run the LogRhythm Database Upgrade Tool from the primary HA server only.

• Upgrade the LogRhythm appliances.
  • Run the LogRhythm Install Wizard on the primary PM/XM.
  • Configure the remaining hosts by running the LogRhythm Infrastructure Installer package on the required appliances.
    • Platform Manager
      • Run the LogRhythm Infrastructure Installer package on the secondary PM/XM via the command line, specifying the /ha-secondary flag: .\LRII_Windows.exe /ha-secondary=<HA shared IP address>
    • Data Processors
    • AI Engine Servers
    • Web Console Servers

    • Data Indexers

      

      If you have Linux Data Indexers (DXs), run the LogRhythm Infrastructure Installer package when upgrading the DX. For more information see Upgrade the LogRhythm Data Indexer.
      • Update the hosts file to contain the hot and warm additions with the host entries.
      • Run the PreInstall.sh script from the DX node you want to run the upgrade from.
      • When prompted to enter the path location for the hosts file, enter the full path location—for example, home/logrhythm/Soft/hosts.

      • Run the installer with the hosts file argument.

        • An --es-cluster-name is required only for new install, not for an upgrade

    • (Optional) Configure additional servers (SOAP API Server).
    • Verify cluster status.
  • Run the LogRhythm Install Wizard on all remaining Windows appliances.
    
    • Secondary PM/XM
      • When the LogRhythm Infrastructure Installer opens, click Exit.
    • Data Processors
    • AI Engine Servers
    • Web Console Servers
  • Run the post-upgrade script on the primary and secondary nodes.
• Perform post-upgrade procedures.
  • Restart the upgraded systems.
  • Import the LogRhythm license file.
  • Start the LogRhythm components.

Verify the Upgrade

• Confirm that all LogRhythm services have started successfully.

• Restart all services that are In Service within the LifeKeeper GUI.

• Confirm that All Services Up appears in the Configuration Manager.

  

  This could take up to 5 minutes after the upgrade.
• Open a web browser on the primary PM/XM, enter <localhost>:3000, click Pipeline, and then click Mediator.
  • Confirm that all Data Processors are processing logs.
  • Change the dashboard time filter to Last 5 minutes.
• Enter <localhost>:3000, click Data Indexer, click Maintenance, and then confirm that the DX cluster is green or yellow.

• 

  • If you have a large deployment, the cluster may remain yellow for a significant amount of time while the indices come online.
  • Depending on the size of the DX cluster and how much data it contains, it could take several minutes for the cluster to turn from red to yellow or green after the upgrade. After it turns yellow, then indexing and searching capabilities should be reinstated.

• Enter <localhost>:3000, click Data Indexer, click Logs Indexing, and then confirm that logs are being indexing into the DX cluster.
• Enter <localhost>:3000, click AIE, click AIE Metrics, and then confirm that all LMs (DPs) are connected to the AIE servers required, and that the AIE servers are receiving and processing data.
• In the Web Console:
  
  • Verify that you can see data on your key dashboards.
  • Conduct a search in the Web Console with the following parameters:
    • Timeframe: Last 30 minutes
    • Filters: remove all filters
    • Repository: Logs

This check tests 90% of the SIEM's core functionality. If you get results, then the deployment's processing, indexing, and searching functionality are working. If you do not get results, then you may need to wait for the DX cluster to turn yellow before trying this search again.

• If you have any test AIE correlation rules set to trigger an alarm, generate one of these and verify that it appears in the Alarms tab of the Web Console.

  

  If you have already had AIE correlation rules trigger an alarm post-upgrade, then there is no need to complete this step

• (Optional) Complete a failover to confirm that the failover works on your new version.

  

  This does not need to be conducted at the same time as the upgrade.