If you have identified NMC database corruption, you can use the NetWorker recoverpsm command to restore the NMC database to a backup of the nmcdb. By default, there should be an NMC server backup workflow inside the default Server Protection policy. Ensure that this is scheduled and configured correctly to complete backups of the NMC database.
For more information about NMC server protection, see the NetWorker Administration Guide and the NetWorker Command Reference Guide for both savepsm and recoverpsm options. These documents are found on the Dell NetWorker Product Page.
mminfo command:mminfo -avot -q client=nmc_server_name
[root@linux1 ~]# mminfo -avot -q client=linux1.amer.lan | grep nmcdb_stage
ClientBackupPool.001 Data Domain linux1.amer.lan 08/15/2022 02:00:17 PM 53 MB 687518690 cb full /nsr/nmc/nmcdb_stage
nmcdb_stage saveset, you cannot recover the NMC and the only option is to reinstall it as a fresh instance.
GSTD service (if running):
Windows: net stop gstd
Linux: systemctl stop gst
Use tasklist/Task Manager or PS commands to ensure that the GSTD service has stopped.
CD command to go to the NMC's installation bin directory. Check to ensure that the recoverpsm command is listed in this directory. This command is used to recover the NMC database.Windows: [NMC Installation Dir\Management\GST\bin
Linux: /opt/lgtonmc/bin
recoverpsm command on the NMC server:recoverpsm [-s server] [-c client-name] [-d destination] [-p pass-phrase] [-t time] [-hfO] Staging_Directory
Options:
|
|
Specifies the name of the source NMC server |
|
|
Specifies the directory to which |
|
|
Use this option to delete the current database files that exist in the database directory before starting the recovery operation. |
|
|
Display the usage information about the |
|
|
Use this option to omit the recovery of the database credential file. Only use this option in the rare case where the credential file is still present and you do not want to use the credential file from the recover. |
|
|
Use this option to specify an additional pass phrase when you attempt to recover AES encrypted database files. See recover (1m) for more details. |
|
|
Specifies the name of the NetWorker server that performed the NMC database backup |
|
|
Use this option to recover the database as of a date specified in |
|
|
Specifies the staging directory used by the backup operation |
Example:
[root@linux1 bin]# ./recoverpsm -f -s linux1.amer.lan -c linux1.amer.lan -p Pa$$w0rd01 /nsr/nmc/nmcdb_stage 108157:recover: Restoring database files... 179040:recover: Step (1 of 11) for PID-6264: Recovery has been started on the client 'linux1.amer.lan'. 179733:recover: Step (2 of 11) for PID-6264: Checking if the set of exclusion files is given for the recovery of the selected files for the client 'linux1.amer.lan'. 179738:recover: Step (3 of 11) for PID-6264: Setting the server 'linux1.amer.lan' for the media database calls for the recovery of the selected files for the client 'linux1.amer.lan'. 179745:recover: Identified a file level recovery with PID 6264 to relocate to the new path '/nsr/nmc/nmcdb' for the client 'linux1.amer.lan'. Updating the total number of steps from 11 to 8. 179748:recover: Step (4 of 8) for PID-6264: Initializing a recovery job with the nsrjobd for the recovery of the selected file(s) for the client 'linux1.amer.lan'. 179749:recover: Step (5 of 8) for PID-6264: Validating and setting the relocation path '/nsr/nmc/nmcdb' for the recovery of selected file(s) for the client 'linux1.amer.lan'. 179754:recover: Step (6 of 8) for PID-6264: Initializing the file level recovery of the client 'linux1.amer.lan'. 179730:recover: Step (7 of 8) for PID-6264: Building the recovery tree of the client 'linux1.amer.lan'. Recovering 1500 files within /nsr/nmc/nmcdb_stage/ into /nsr/nmc/nmcdb Requesting 1500 file(s), this may take a while... 179728:recover: Initializing the recovery session with the NetWorker server 'linux1.amer.lan' to recover the selected files. Recover start time: Mon 15 Aug 2022 02:20:37 PM PDT 179718:recover: Initializing to recover multiple streams of data from the NetWorker server 'linux1.amer.lan'. Requesting 1 recover session(s) from server. 179714:recover: Initializing the child threads to recover the given recovery stream from the NetWorker server 'linux1.amer.lan'. 179713:recover: Initializing to set up a connection to start reading the data from the recover stream. Enabling compressed restore for save set ID '687518690' with 'Data Domain' device path '/linux1/ClientBackupDevice01'. Successfully established the direct file retrieval session for save set ID '687518690' with 'Data Domain' volume 'ClientBackupPool.001'. ./pgdata/backup_label ... ./pgdata/ 179716:recover: Coordinating all the child recovery processes. 179717:recover: Releasing all the resources held by the recovery process. 179721:recover: All recovery streams from the NetWorker server 'linux1.amer.lan' are read successfully. Received 1500 file(s) from NSR server `linux1.amer.lan' Recover completion time: Mon 15 Aug 2022 02:20:39 PM PDT 179068:recover: Step (8 of 8) for PID-6264: The recovery completed successfully on the client 'linux1.amer.lan'. 108161:recover: Setting permissions for the restored database files
GSTD service:net start gstd
systemctl start gst
tasklist/Task Manager or PS commands to ensure that the GSTD service has started.
/opt/lgtonmc/logs/gstd.raw
[Install Drive]:\Program Files\EMC NetWorker\Management\GST\logs\gstd.raw
The NMC server's ..\Management\GSTD\logs\gstd.raw or /opt/lgtonmc/logs/gstd.raw can be used to confirm if there is NMCDB corruption:
Example:
31581 1501512324 2 0 0 5632 6204 0 NMC-server gstd NSR warning 23 *** gstd starting at %s 1 36 24 Mon Jul 31 10:45:24 2017
NMC-server gstd NSR notice 5 %s %s 2 0 24 MM/DD/YYY HH:MM:SS.3852 0 50 Internal error: could not create database process.
NMC-server gstd NSR notice 5 %s %s 2 0 24 MM/DD/YYY HH:MM:SS.643852 0 50 Internal error: could not create database process.
NMC-server gstd NSR notice 5 %s %s 2 0 24 MM/DD/YYY HH:MM:SS.643852 0 50 Internal error: could not release database handle.
NMC-server gstd NSR notice 5 %s %s 2 0 24 MM/DD/YYY HH:MM:SS.675017 0 48 Internal error: could not stop database process.
NMC-server gstd NSR notice 5 %s %s 2 0 24 MM/DD/YYY HH:MM:SS.675017 0 54 FATAL ERROR: could not stop 'EMC GST Database Service'
NMC-server gstd NSR notice 5 %s %s 2 0 24 MM/DD/YYY HH:MM:SS.675017 0 48 Internal error: could not stop database process.
NMC-server gstd NSR notice 5 %s %s 2 0 24 MM/DD/YYY HH:MM:SS.675017 0 48 Internal error: could not stop database process.
NMC-server gstd NSR notice 5 %s %s 2 0 24 MM/DD/YYY HH:MM:SS.675017 0 54 The most recently called subfunction returned error 2.
NMC-server gstd NSR warning 2 %s 1 0 27 library error 22 os error 0
NMC-server gstd NSR warning 23 *** gstd stopping at %s 1 36 24 MM/DD/YYY HH:MM:SS