PowerProtect Data Manager 19.9 Administration and User Guide

Error displays during node failure

The following error might display during a search when a node fails:

Not able to deploy search-node.com. Another session "<host_name>" is already configured with the same hostname. Would you like to redeploy search node or delete the node?

If this error occurs, delete the node, and then retry the operation. If you choose to edit, delete the node and the new mode modal appears with your previous input. The input that caused the error is marked as critical.

Certificate issues

Issues with indexing backups and/or performing search queries might result when certificates that were deployed on the search node were corrupted.

Perform one of the following tests to determine certificate issues:

• Use the log bundle download utility in PowerProtect Data Manager to examine the Backup VM logs in VM Direct, and look for a log entry like the following:

  ERROR: Failed to Upload File: /opt/emc/vproxy/runtime/tmp/vproxyd/
  plugin/search/e6c356a1-fbaf-4231-9f6f-a0166b74909a/<search 
  node>-e081fdea-3599-4a6c-abc4-1b5487cb9a32-e523a94c-2d01-5234-ab3c-
  7771cfab3c58-7f16bcbb72d7b49ea073356f0d7388ac08461827.db.zip to 
  https://<search node>:14251/upload, Error sending data chunk. Post 
  https://<search node>:14251/upload: x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "PPDM Root CA ID-d5ec56b8-69ec-4183-9c94-7c0230408765"

• Examine the rest-engine logs in the search node (/opt/emc/search/logs/rest-engine/*.log), and look for certificate verification errors.
• Run a search either through the UI or through the API <PowerProtect Data Manager>/api/v2/file-instances and look for a certification verification error.

Examine the certificate files in the node(s) to investigate further. If necessary, regenerate the certificate files.

Access the Search Node to discover passwords

Use the following steps to discover the admin and root passwords for all deployed search nodes:

1. Connect to the PowerProtect Data Manager console and change to the root user.
2. Change directory to /opt/emc/vmdirect.
3. Source unit/vmdirect.env.
4. Run bin/infranodemgmt get -secret.

Verify certificates

Use this procedure to verify that certificates are valid and uncorrupted:

1. Verify that the rootca.pem file is the same in all the relevant nodes (search node, PowerProtect Data Manager, and VM Direct node).
   NOTE The rootca.pem file name is different on each node:

   • PowerProtect Data Manager— /etc/ssl/certificates/rootca/rootca.pem
   • Search node— /var/lib/dellemc/vmboot/trust/thumbprint
   • VM Direct— /var/lib/dellemc/vmboot/trust/thumbprint
2. Run the following openssl command to find out whether the root certificate file is corrupt or invalid: openssl verify <rootca.pem>

   Response:  

   /var/lib/dellemc/vmboot/trust/thumbprint: C = US, 
   O = DELL Corporation, 
   CN = PPDM Root CA ID-4c9de850-24ab-42ec-a9a7-6080849d0d24
   
   error 18 at 0 depth lookup:self signed certificate
   
   
   OK

   Ensure that the CN values match.

Certificate verification fails

If the certificate verification steps fail, you must re-create the certificates on the Search Node or VM Direct node:

1. Connect to the PowerProtect Data Manager console and change to the root user.
2. Use the Get command in the infranodemgmt utility to determine the search node FQDN.
3. Run /usr/local/brs/puppet/scripts/generate_certificates.sh -n -c -b <search node FQDN>

   A properties file is created in the /root directory called <search node FQDN>.properties.  

4. Open this file to determine the location of the generated certificates. They should be located in /etc/ssl/certificates/<search node FQDN>.
5. From a separate terminal, SSH into the search node using the password that was revealed with the infranodemgmt Get call in step 2.
6. Change directory to /var/lib/dellemc/vmboot/trust and move the key, cert, and thumbprint files over.
7. Copy the certificate files that were generated in PowerProtect Data Manager as follows:
   • otca.pem to thumbprint
   • <search node FQDN>key.pem to key
   • <search node FQDN>.pem to cert
8. Paste the files to /var/lib/dellemc/vmboot/trust.
9. Set the permissions for the key, cert, and thumbprint files to 0644, and then set the ownership of these files to root:app
10. Restart the rest-engine daemon or the vproxyd daemon) to pick up the new certificates: systemctl restart search-rest-engine.
11. Check the rest-engine log file (/opt/emc/search/logs/rest-engine/rest-engine-daemon-<fqdn>.log) to verify that the service started successfully.

    Ensure that the following message appears:  

    A valid Root CA certificate of backup server was provided during deployment
    

Result: Backup with indexing executes successfully and search service is functional.

Search cluster is full

If the search cluster is full, you can deploy additional nodes by following the steps in Set up and manage indexing.

If the search cluster runs out of space and you do not want to deploy an additional node, you have the following options:

• Disable the service
• Shorten the expiration time to remove indexes sooner
• Remove indexes manually

To disable the service, complete the following steps:

1. From the PowerProtect Data Manager UI, select Infrastructure > Search Engine.
2. Select the cluster, and then click Configure Cluster.
3. In the Configure Search Cluster dialog box, switch the Search Indexing button to turn it off, and then click Save.
   NOTE This setting applies to all indexes in all protection policies in the Search Cluster.

To shorten the expiration time to remove indexes sooner, complete the following steps:

1. From the PowerProtect Data Manager UI, select Infrastructure > Search Engine.
2. Select the cluster, and then click Configure Cluster.
3. In the Configure Search Cluster dialog box, modify the Search Index Expiration and click Save. A recommended formula to determine the expiration time is: Delete Index when Today = Backup-Date + Expiration Days + 1 day. That is, one day after the backup expires.
   NOTE This setting applies to all indexes in all protection policies in the Search Cluster.

To remove indexes manually, complete the following steps:

1. Use SSH to log in to the Search virtual machine.
2. Create a snapshot of the Search cluster using the following format:

           {
               Command:  "APP_SNAPSHOT",
               Title:    "Initiate Index/Search Cluster Snapshot Process",
               AsyncCmd: false,
               Properties: {
                   "Name": {
                       Description: "Used to uniquely identify a particular snapshot",
                       Type:        STRING
                   },
                   "Action": {
                       Description: "Action to perform, 'Create', 'Delete', 'Restore' or 'Cancel' a Snapshot",
                       Type:        STRING
                   },
                   "NFSHost": {
                       Description: "NFS Host serving snapshot backup area.",
                       Type:        STRING
                   },
                   "NFSExport": {
                       Description: "NFS Export path to mount too.",
                       Type:        STRING
                   },
                   "NFSDirPath": {
                       Description: "NFS directory path to write too.",
                       Type:        STRING
                   }
               }
           }

   For example:  

   {
   	"Command": "APP_SNAPSHOT",
   	"Title": "",
   	"AsyncCmd": false,
   	"Properties": {
   		"Action": {
   			"Description": "",
   			"Required": false,
   			"Type": "string",
   			"IsArray": false,
   			"Value": "Create",
   			"Default": null
   		},
   		"Name": {
   			"Description": "",
   			"Required": false,
   			"Type": "string",
   			"IsArray": false,
   			"Value": "DataManager_Catalog_Cluster_snapshot_2019-10-16-12-57-16",
   			"Default": null
   		},
   		"NFSHost": {
   			"Value": "10.25.87.88"
   		},
   		"NFSExport": {
   			"Value": "/mnt/shared"
   		},
   		"NFSDirPath": {
   			"Value": ""
   		}
   	}
   } 

3. You can delete indexes by protection policy or by asset. If the JSON command is stored at /home/admin/remove-plc.json, run the command, ./searchmgmt -I /home/admin/remove-plc.json.
   • Use the following format to delete indexes by protection policy:

     {
                 "Command": "APP_REMOVE_ITEMS",
                 "AsyncCmd": false,
                 "Properties": {
                             "Action": {
                                         "Description": "Action to perform, 'AssetDelete', 'PLCDelete'",
                                         "Required": true,
                                         "Value": "PLCDelete",
                             }
                             "PLCID": {
                                         "Description": "PLC ID of item(s) to delete.",
                                         "Required": true,
                                         "Value": "7676d753-b57e-a572-6daf-33689933456d",
                             }
                 }
     }

   • Use the following format to delete indexes by asset type:

     {
                 "Command": "APP_REMOVE_ITEMS",
                 "AsyncCmd": false,
                 "Properties": {
                             "Action": {
                                         "Description": "Action to perform, 'AssetDelete', 'PLCDelete'",
                                         "Required": true,
                                         "Value": "AssetDelete",
                             },
                             "AssetID": {
                                         "Description": "Optional, Asset ID of item(s) to delete.",
                                         "Required": false,
                                         "Value": "503dd753-b57e-a572-6daf-44680033755f",
                             },
                             "PLCID": {
                                         "Description": "PLC ID of item(s) to delete.",
                                         "Required": true,
                                         "Value": "7676d753-b57e-a572-6daf-33689933456d",
                             }
                 }
     }

   NOTE

   • The time to complete the execution of these procedures depends on the number of backup copy asset indexes being deleted.
   • This procedure does not impact regular operation of the cluster.

Troubleshooting a locked Search Engine Node

The nodes on the PowerProtect Data Manager Search cluster are configured with IP addresses that can be accessed externally. These nodes are configured with admin or root user accounts, which are only used to log in to the Search nodes for troubleshooting software issues. The password management policies for these accounts are set to lock the admin user account if there are three wrong password attempts within a 5 minute time period. If you try to access the node while the admin user account is locked, the amount of time that the account remains locked increases.

There is no public interface available that enables you to access the search node by using admin credentials. All required information about the Search Engine nodes is obtained through the PowerProtect Data Manager UI.

A Search node might become locked for the following reasons:

• A user or program tries to SSH into the search node and makes three wrong attempts at entering the password.
• Running monitoring software that tries to log in to the Search node with the wrong admin credentials and locks the system.
• Running Penetration Testing (PEN) on the VMs in the vCenter.

The Search admin user account enables the PowerProtect Data Manager system to perform different operations on the Search node, such as obtaining the health status of the node. If the account is locked, the health status of the node is reported as "Failed." When one of the nodes in the Search cluster is in a failed state, the entire Search cluster becomes unavailable. As a result, the Search cluster is unable to perform any indexing or search operations.

Workaround

To work around this issue, first access the Search node to discover the admin and root credentials for the node. After you discover the node credentials, log in to the node through the vCenter console to reset the admin credentials.

Use the following steps to discover the admin and root passwords for all deployed Search nodes:

1. Connect to the PowerProtect Data Manager console and change to the root user.
2. Change directory to /opt/emc/vmdirect
3. Run unit/vmdirect.env
4. Run bin/infranodemgmt get -secret

Before you access the Search node through the vCenter console, determine why the admin account is locked.

Use the following steps to unlock the admin user account:

1. Log in to the vCenter where the Search node is present.
2. Select the Search node from the VMs and Templates view in the left pane of the vSphere Client home page.
3. Launch a PowerProtect Data Manager VM vCenter console.
4. Log in to the vCenter console with the root username and password.
5. Run the following command to reset the admin user account credentials:

   /sbin/pam_tally2 --user admin --reset