NetWorker - How to Recover the NetWorker Management Console database

If you have identified NMC database corruption, you can use the NetWorker recoverpsm command to restore the NMC database to a backup of the nmcdb. By default, there should be an NMC server backup workflow inside the default Server Protection policy. Ensure that this is scheduled and configured correctly to complete backups of the NMC database.

For more information about NMC server protection, see the NetWorker Administration Guide and the NetWorker Command Reference Guide for both savepsm and recoverpsm options. These documents are found on the Dell NetWorker Product Page.

1. Ensure that there is a backup of the NMC database using the mminfo command:

[root@linux1 ~]# mminfo -avot -q client=linux1.amer.lan | grep nmcdb_stage
ClientBackupPool.001 Data Domain linux1.amer.lan 08/15/2022 02:00:17 PM 53 MB 687518690 cb full /nsr/nmc/nmcdb_stage

2. On the NMC server, open an Administrative root command prompt to stop the NMC GSTD service (if running):

Windows: net stop gstd
Linux: systemctl stop gst

Use tasklist/Task Manager or PS commands to ensure that the GSTD service has stopped.

3. Use the CD command to go to the NMC's installation bin directory. Check to ensure that the recoverpsm command is listed in this directory. This command is used to recover the NMC database.

Windows: [NMC Installation Dir\Management\GST\bin
Linux: /opt/lgtonmc/bin
 

4. Run the recoverpsm command on the NMC server:

recoverpsm [-s server] [-c client-name] [-d destination] [-p pass-phrase] [-t time] [-hfO] Staging_Directory

Options:

Example:

[root@linux1 bin]# ./recoverpsm -f -s linux1.amer.lan -c linux1.amer.lan -p Pa$$w0rd01 /nsr/nmc/nmcdb_stage
108157:recover: Restoring database files...
179040:recover: Step (1 of 11) for PID-6264: Recovery has been started on the client 'linux1.amer.lan'.
179733:recover: Step (2 of 11) for PID-6264: Checking if the set of exclusion files is given for the recovery of the selected files for the client 'linux1.amer.lan'.
179738:recover: Step (3 of 11) for PID-6264: Setting the server 'linux1.amer.lan' for the media database calls for the recovery of the selected files for the client 'linux1.amer.lan'.
179745:recover: Identified a file level recovery with PID 6264 to relocate to the new path '/nsr/nmc/nmcdb' for the client 'linux1.amer.lan'. Updating the total number of steps from 11 to 8.
179748:recover: Step (4 of 8) for PID-6264: Initializing a recovery job with the nsrjobd for the recovery of the selected file(s) for the client 'linux1.amer.lan'.
179749:recover: Step (5 of 8) for PID-6264: Validating and setting the relocation path '/nsr/nmc/nmcdb' for the recovery of selected file(s) for the client 'linux1.amer.lan'.
179754:recover: Step (6 of 8) for PID-6264: Initializing the file level recovery of the client 'linux1.amer.lan'.
179730:recover: Step (7 of 8) for PID-6264: Building the recovery tree of the client 'linux1.amer.lan'.
Recovering 1500 files within /nsr/nmc/nmcdb_stage/ into /nsr/nmc/nmcdb
Requesting 1500 file(s), this may take a while...
179728:recover: Initializing the recovery session with the NetWorker server 'linux1.amer.lan' to recover the selected files.
Recover start time: Mon 15 Aug 2022 02:20:37 PM PDT
179718:recover: Initializing to recover multiple streams of data from the NetWorker server 'linux1.amer.lan'.
Requesting 1 recover session(s) from server.
179714:recover: Initializing the child threads to recover the given recovery stream from the NetWorker server 'linux1.amer.lan'.
179713:recover: Initializing to set up a connection to start reading the data from the recover stream.
Enabling compressed restore for save set ID '687518690' with 'Data Domain' device path '/linux1/ClientBackupDevice01'.
Successfully established the direct file retrieval session for save set ID '687518690' with 'Data Domain' volume 'ClientBackupPool.001'.
./pgdata/backup_label
...
./pgdata/
179716:recover: Coordinating all the child recovery processes.
179717:recover: Releasing all the resources held by the recovery process.
179721:recover: All recovery streams from the NetWorker server 'linux1.amer.lan' are read successfully.
Received 1500 file(s) from NSR server `linux1.amer.lan'
Recover completion time: Mon 15 Aug 2022 02:20:39 PM PDT
179068:recover: Step (8 of 8) for PID-6264: The recovery completed successfully on the client 'linux1.amer.lan'.
108161:recover: Setting permissions for the restored database files

5. On the NMC server, open an Administrative root command prompt to stop the NMC GSTD service:

6. Ensure that you can connect to the NMC.

7. Monitor the gstd.raw for any errors during NMC service startup:

The NMC server's ..\Management\GSTD\logs\gstd.raw or /opt/lgtonmc/logs/gstd.raw can be used to confirm if there is NMCDB corruption:

Example:

31581 1501512324 2 0 0 5632 6204 0 NMC-server gstd NSR warning 23 *** gstd starting at %s 1 36 24 Mon Jul 31 10:45:24 2017
NMC-server gstd NSR notice 5 %s %s 2 0 24 MM/DD/YYY HH:MM:SS.3852 0 50 Internal error: could not create database process.
NMC-server gstd NSR notice 5 %s %s 2 0 24 MM/DD/YYY HH:MM:SS.643852 0 50 Internal error: could not create database process.
NMC-server gstd NSR notice 5 %s %s 2 0 24 MM/DD/YYY HH:MM:SS.643852 0 50 Internal error: could not release database handle.
NMC-server gstd NSR notice 5 %s %s 2 0 24 MM/DD/YYY HH:MM:SS.675017 0 48 Internal error: could not stop database process.
NMC-server gstd NSR notice 5 %s %s 2 0 24 MM/DD/YYY HH:MM:SS.675017 0 54 FATAL ERROR: could not stop 'EMC GST Database Service'
NMC-server gstd NSR notice 5 %s %s 2 0 24 MM/DD/YYY HH:MM:SS.675017 0 48 Internal error: could not stop database process.
NMC-server gstd NSR notice 5 %s %s 2 0 24 MM/DD/YYY HH:MM:SS.675017 0 48 Internal error: could not stop database process.
NMC-server gstd NSR notice 5 %s %s 2 0 24 MM/DD/YYY HH:MM:SS.675017 0 54 The most recently called subfunction returned error 2.
NMC-server gstd NSR warning 2 %s 1 0 27 library error 22 os error 0
NMC-server gstd NSR warning 23 *** gstd stopping at %s 1 36 24 MM/DD/YYY HH:MM:SS