After initiating an Avamar recovery in the
Cyber Recovery UI, perform the following steps on the Avamar server host in the
Cyber Recovery vault.
Prerequisites
You have successfully created a Recovery Sandbox in Cyber Recovery.
There is an executable copy of the
lockbox_restore.pl script downloaded from
ftp://avamar_ftp:anonymous@ftp.avamar.com/software/scripts/lockbox_restore.pl in the
/home/admin/ directory on the Avamar server in the
Cyber Recovery vault. The script must be owned by the
admin user.
You have the required credentials:
Component
Description
Application
Login credentials for PuTTY and Avamar
Avamar
The admin and root user accounts, which might be stored on a specific system or in a document
DD Boost
Avamar
DD Boost user id and password on the
Cyber Recovery vault
Cyber Recovery username
cradmin
About this task
Use a PuTTY or SSH session that is connected to the Avamar server in the
Cyber Recovery vault to perform the following procedure.
Steps
Use PuTTY to log in to the Avamar server as the
admin user.
Stop the Avamar services:
dpnctl stop all
Answer
Yes to the query about shutting down the local instance of EM Tomcat.
Confirm that the services are stopped properly:
dpnctl status
Switch to the Avamar
root user:
su -
Verify that the IP address of the vault DD system resolves to the production DD name. This step ensures that the Avamar server can connect to the vault DD system and perceives it as the production DD system in the vault. Modify the Avamar
/etc/hosts file on the
Cyber Recovery vault for both Avamar and Data Domain, as needed.
CAUTION:This step is critical for performing the recovery.
In the following example,
ddve-prod-05 is the name of the production DD system and 192.168.2.106 is the IP address of the vault DD system (also known as
ddve-cr-06). Both the DD FQDNs and short names are assigned to the 192.168.2.106 IP address.
cat /etc/hosts
127.0.0.1 localhost.localdomain localhost
::1 localhost.localdomain localhost
#(ave-03 is the production hostname
# but the IP specified must point to the vault IP.)
192.168.2.83 ave-03.vcorp.local ave-03
192.168.2.106 ddve-prod-05.vcorp.local ddve-prod-05 ddve-cr-06.vcorp.local ddve-cr-06
NOTE:The following FQDN names are used in examples throughout the rest of this document:
A list of available checkpoints that can be used to restore is displayed. The checkpoint at the bottom of the list is the most recent checkpoint.
Enter the name of the checkpoint that you want to restore, for example
cp.20211216090102, ensuring that the name is exact. Press Enter and when prompted, type
yes to confirm your entry.
NOTE:This step might take some time as it is copying data from the vault DD system to the staging Avamar server to perform the recovery steps. Press Enter every few minutes in the PuTTY window to avoid timing out.
When the checkpoint restore operation completes, enter the DD Boost user password when you are prompted for a password.
Switch to the Avamar
admin user:
su - admin
Start the Avamar rollback procedure using the same checkpoint name as the checkpoint name that you selected for the checkpoint restore operation:
rollback.dpn --cptag=cp.20211216090102 --noddrollback --nogetserverlogs 2>&1 | tee -a rollback.out
This step can take some time. When it is completed, it displays the output of the
status.dpn command.
As the
admin user, list the hostname of the Avamar server:
hostname -f
Begin the rollback of the MCS services:
mcserver.sh --restore --norestart --v 2>&1 |tee -a mcs_restore.out
When prompted, enter
Y to proceed with the restore, enter the
<Production_Stager_Avamar-FQDN> obtained from the
hostname -f command; press Enter for port 27000.
Switch to the Avamar
root user:
su -
Run the
lockbox_restore.pl
script and answer
yes to all the prompts:
/home/admin/lockbox_restore.pl
NOTE:If an error is displayed for the lockbox, type
yes to proceed and then provide the correct operating system password for the
admin user.
For example:
Sample run updating “admin”: (Note – this run entered a BAD password the first time. Second time was successful):
Your keystore contains 4 entries
Keystore certs: [mcectls, Nov 10, 2021] [mcrsatls, Nov 10, 2021] [mcecroot, Nov 10, 2021] [mcrsaroot, Nov 10, 2021]
DEBUG: Checking lockbox 'admin' key...
Sorry, try again.
Sorry, try again.
sudo: 3 incorrect password attempts
ERROR: 'sudo -A' failed. Error=256
This indicates a problem with the 'admin' password stored in the lockbox.
This will cause downstream problems with MCS startup.
Lockbox verification FAILED for admin. Proceed ?
Enter `yes`<enter> to proceed, `q` to quit :yes
[LOCKBOX] Enter New lockbox entry for 'admin':*********
>>Backup lockbox file
>>Backup keystore files
>>Backup SSV files
>>Flush backup
>>Local backup dir: /usr/local/avamar/src/lockbox_backup/2022-05-24-20_33
>>Flush backup dir: /usr/local/avamar/var/mc/server_data/lockbox_backup
>>Updated with new value under name "admin".
>>Backup lockbox file
>>Backup keystore files
>>Backup SSV files
>>Flush backup
>>Local backup dir: /usr/local/avamar/src/lockbox_backup/2022-05-24-20_33
>>Flush backup dir: /usr/local/avamar/var/mc/server_data/lockbox_backup
Sorry, try again.
Sorry, try again.
sudo: 3 incorrect password attempts
ERROR: 'sudo -A' failed. Error=256
This indicates a problem with the 'admin' password stored in the lockbox.
This will cause downstream problems with MCS startup.
Lockbox verification FAILED for admin. Proceed ?
Enter `yes`<enter> to proceed, `q` to quit :yes
[LOCKBOX] Enter New lockbox entry for 'admin':**************
>>Backup lockbox file
>>Backup keystore files
>>Backup SSV files
>>Flush backup
>>Local backup dir: /usr/local/avamar/src/lockbox_backup/2022-05-24-20_33
>>Flush backup dir: /usr/local/avamar/var/mc/server_data/lockbox_backup
>>Updated with new value under name "admin".
>>Backup lockbox file
>>Backup keystore files
>>Backup SSV files
>>Flush backup
>>Local backup dir: /usr/local/avamar/src/lockbox_backup/2022-05-24-20_33
>>Flush backup dir: /usr/local/avamar/var/mc/server_data/lockbox_backup
DEBUG: Avagent version: 19.4.100-116
DEBUG: Avagent OS version: SLES-64
Switch to the Avamar
admin user:
su - admin
Start the MCS:
mcserver.sh --start --v 2>&1 |tee -a mcs_start.out
Verify the services are up and running:
dpnctl status
Start any subsystems that are stopped:
dpnctl start <subsystem>
Ensure that the
emt and
ddrmaint-service are started:
dpnctl start emt
dpnctl start ddrmaint-service
NOTE:Leave the scheduler and maintenance processes as
down.
Switch to the Avamar
root user:
su -
Add the SSH key for the Data Domain FQDN, using the following syntax:
NOTE: If the value for the first two options is
false, type
enable_secure_config.sh --enable-secure-all and then type
enable_secure_config.sh --showconfig to check the security settings again.
Switch to the Avamar
admin user:
su - admin
Restart the MCS:
mcserver.sh --restart --v 2>&1 |tee -a mcs_start.out
Enter
Y to proceed.
Edit the DD properties:
mccli dd edit --name=<Production_DD-FQDN>
Confirm the DD properties:
mccli dd show-prop --name=<Production_DD-FQDN>
This step takes several minutes as it edits the DD name in the MCS. When the step is completed, the DD
<Production_DD-FQDN> is displayed in several lines.
Switch to the Avamar
root user:
su -
Revoke the token access using the following syntax:
ssh cradmin@<Production_DD-FQDN> "ddboost user revoke token-access <ddboost username>"
For example:
ssh sysadmin@ddve-prod-05.vcorp.local"ddboost user revoke token-accessddboostuser"
Enter the password for the sysadmin.
NOTE: This command can use the sysadmin or cradmin user to revoke the token access. The command output displays the following message:
To take a checkpoint and validate it, type the following commands:
dpnctl start ddrmaint-servicedpnctl stop maintmcserver.sh --flushavmaint checkpoint --ava<Wait a few minutes while the checkpoint is being created.>cplist --lscp<A new checkpoint is displayed based on the current date.>
Log in to the Avamar UI using the MCUser on the Avamar host server (https://<avamar-host>/aui ). From the left navigation pane, go to
Administration > System and then select
Data Domain on the right pane.
Verify that the DD system is displayed in the main window.
Verify that the data represented on the DD properties matches the data of the Avamar DD system. The icons that precede the entry must be green or at least amber.
From the Avamar navigation menu options, verify that all the policies, clients, and other configuration items match those items of the production system.
Return to PuTTY to ensure that the hfscheck procedure is completed and the status is
complete. Press Ctrl-c to exit PuTTY.
See Avamar's standard operating procedures to reactivate clients in the
Cyber Recovery vault and perform the required application recoveries.