Disaster Recovery Failover and Fallback

If you need to designate the Secondary site as the Active site, read this section for an overview of the failover process and for detailed instructions on how to switch between sites.

Overview of the Failover Process

The failover process depends on whether you purposely shut down the Primary site (Planned Failover) or the Primary site went down unexpectedly (Unplanned Failover). In either case, you must manually perform a failover, as outlined below:

1. Manually initiate the failover, using the appropriate process:

   • Planned Failover. Go to the Primary (active) site and use the DR Control (DR_Monitoring.ps1) script to initiate the failover to the standby site. For more information, see Perform a Planned Failover below.

   • Unplanned Failover. Go to the Secondary (standby) site and use the DR Control (DR_Monitoring.ps1) script to initiate the failover to the standby site. For more information, see Perform an Unplanned Failover below.

2. Update the shared DNS record so the Primary site components point to the IP address of the Secondary Platform Manager. Once the time to live (TTL) limit is reached, all systems in the Primary site reconnect to the newly activated Platform Manager.

3. If a Data Processor is unavailable on the Primary site, reconnect Agents to a new Data Processor by changing the DNS records.

Distinction between DR and HA+DR

A DR-enabled deployment provides protection for the SQL databases and Platform Manager services only. The deployment consists of a Primary site and a DR site, with all other platform components expected to operate in an active/active configuration. During a failover, the active workload is transferred from the Primary site to the DR site.

In an HA+DR deployment, the environment consists of three nodes: HA1, HA2, and the DR site. Recovery is performed in a tiered manner. Local High Availability failover between HA1 and HA2 is always attempted first, with the DR site serving as the final recovery target when local HA protection is unavailable.

High Availability protects the active Platform Manager (or XM) and all dependent services contained within its hierarchy. The DR configuration provides protection only for resources included in the DR hierarchy and is designed to facilitate recovery from a site-wide outage.

As a result, HA+DR and DR-only deployments follow different failover recovery workflows.

Perform a Planned Failover - Disaster Recovery Only

1. Access the Primary (active) Platform Manager.

2. Click Start, All Programs, LogRhythm, and Disaster Recovery.

3. Right-click DR Control and click Run as administrator. Enter your local system administrator credentials.

4. To display the DR Control Options, press D.

5. To initiate the failover process, type F.

6. At the Would you like to failover… prompt, type Y.
   The DR solution automatically performs the following tasks:

   1. Stops the Platform Manager services on the Primary (active) site.

   2. Makes sure all databases are in sync between the Primary and Secondary sites.

   3. Designates the Secondary Platform Manager as the Active site. In DR Controls, the Role column displays Standby.

7. Update the DNS record so that all components point to the IP address of the Secondary Platform Manager.
   After the time to live (TTL) limit is reached, all systems reconnect to the newly activated Platform Manager.

8. Go to the Secondary Platform Manager server and confirm that the Platform Manager services, which include the Alarming and Response Manager (ARM) service and the Job Manager service, have started. If necessary, also start the services for the Data Processors, Data Indexers, and the AI Engines.

9. If necessary, reconnect remote systems to the Secondary Platform Manager by changing the DNS records. If a Data Processor is unavailable, reconnect Agents to a new Data Processor by changing the DNS records or by using the Deployment Manager in the SIEM Console to redirect them.

Perform a Planned Failover - HA + DR Environment

1. Access the Primary (active) Platform Manager.

2. Load LifeKeeper (Admin Only) by right clicking, selecting Run as administrator, and logging in with Administrator credentials.

3. Right-click the top-level ResTag at the top of the Hierarchy and select Out of Service…

4. Select the local Server > Next > Out of Service.

5. Click Start, All Programs, LogRhythm, and Disaster Recovery.

6. Right-click DR Control and click Run as administrator.

7. Enter your local system administrator credentials.

8. To display the DR Control Options, press D.

9. To initiate the failover process, type F.

10. At the Would you like to failover… prompt, type Y.
    The DR solution automatically performs the following tasks:

    1. Stops the Platform Manager services on the Primary (active) site.

    2. Makes sure all databases are in sync between the Primary and Secondary sites.

    3. Designates the Secondary Platform Manager as the Active site. In DR Controls, the Role column displays Standby.

11. Update the DNS record so that all components point to the IP address of the Secondary Platform Manager.
    After the time to live (TTL) limit is reached, all systems reconnect to the newly activated Platform Manager.

12. Go to the Secondary Platform Manager server and confirm that the Platform Manager services, which include the Alarming and Response Manager (ARM) service and the Job Manager service, have started.
    If necessary, also start the services for the Data Processors, Data Indexers, and the AI Engines.

13. If necessary, reconnect remote systems to the Secondary Platform Manager by changing the DNS records.
    If a Data Processor is unavailable, reconnect Agents to a new Data Processor by changing the DNS records or by using the Deployment Manager in the SIEM Console to redirect them.

Perform an Unplanned Failover - Disaster Recovery Only

1. Go to the Secondary (standby) Platform Manager.

2. Click Start, All Programs, LogRhythm, and Disaster Recovery.

3. Right-click DR Control and click Run as administrator. Enter your local system administrator credentials.
   The DR solution displays a warning, indicating that executing a failover from the standby system may result in data loss. Because the Primary site may have gone offline before the databases were fully synchronized, some data may be lost.

4. To continue, type Y.
   The DR solution automatically performs the following tasks:

   1. Switches the Secondary Platform Manager to the Active state.

   2. Starts the Platform Manager services on the Secondary site.

   3. Loads the replicated databases. In the DR Controls, the Role column displays “Active,” and the State column is still “Disconnected” to indicate the databases are not currently replicating.

5. When the failover is complete, press Enter to exit.

 Perform an Unplanned Failover - HA + DR Environment

1. Go to the Secondary (standby) Platform Manager.

2. Click Start, All Programs, LogRhythm, and Disaster Recovery.

3. Right-click DR Control and click Run as administrator. Enter your local system administrator credentials.

4. To display the DR Control Options, type D.

5. To initiate the failover, type F.
   The DR solution displays a warning, indicating that executing a failover from the standby system may result in data loss. Because the Primary site may have gone offline before the databases were fully synchronized, some data may be lost.

6. To continue, type Y.
   The DR solution automatically performs the following tasks:

   1. Switches the Secondary Platform Manager to the Active state.

   2. Starts the Platform Manager services on the Secondary site.

   3. Loads the replicated databases.

   In the DR Controls, the Role column displays Active, and the State column is still Disconnected to indicate the databases are not currently replicating.

7. Update the DNS record so that all components point to the IP address of the Secondary Platform Manager.
   After the Time to Live (TTL) limit is reached, all systems within the Primary site reconnect to the newly activated Platform Manager.

8. If necessary, reconnect remote systems to the Secondary Platform Manager by changing the DNS records. If a Data Processor is unavailable on the Primary site, reconnect Agents to a new Data Processor by changing the DNS records or by using the Deployment Manager in the SIEM Console.

Resume Operations on the Primary Platform Manager

When the Primary Platform Manager becomes operational again after a failover to the DR site, replication can be resumed and the active role returned to the Primary site. For HA+DR deployments, High Availability protection must also be restored after the failback process is complete.

1. Access the Primary Platform Manager.

2. Open DR Control as an administrator.

3. Verify the State column displays Suspended, indicating the Primary site is available and ready for replication to resume.

4. Press D to display the DR Control options, then press R to resume replication.

5. Wait for all databases to reach a Synchronized state.

6. From the Primary Platform Manager, initiate a planned failback by selecting F from the DR Control menu and confirming the operation.

7. Verify the Primary Platform Manager is reporting the Active role and all Platform Manager services have started successfully.

8. Update DNS records as required so dependent systems reconnect to the Primary Platform Manager.

Additional Steps for HA+DR Deployments

1. Open the LifeKeeper GUI and ensure the top level service (e.g. PM_ResTag, XM_ResTag) is declared as In Service on the local site.
   This is intentionally stopped during a planned failover when the site is available to prevent conflicts between HA and DR sites.

2. Verify hierarchy ownership resides on the intended active HA node and that all protected resources display an operational status.

3. Confirm High Availability protection has been fully restored before considering recovery complete.