PowerProtect Data Manager 19.12 Administration and User Guide

Some Search Engine troubleshooting procedures require the credentials for individual Search Engine nodes. Search Engine nodes have admin and root user accounts that are used for troubleshooting software issues. The PowerProtect Data Manager Security Configuration Guide provides instructions to manage Search Engine node credentials.

Error displays during Search Engine node failure

The following error might display during a search when a Search Engine node fails:

Not able to deploy search-node.com. Another session "<host_name>" is already configured with the same hostname. Would you like to redeploy search node or delete the node?

If this error occurs, delete the Search Engine node, and then retry the operation. If you choose to edit, delete the node. The new mode modal then appears with your previous inputs. The input that caused the error is marked as critical.

Certificate issues

Issues with indexing backups and/or performing search queries might result when certificates that were deployed on the Search Engine node were corrupted.

Perform one of the following tests to determine certificate issues:

• Use the log bundle download utility in PowerProtect Data Manager to examine the Backup VM logs in VM Direct, and look for a log entry like the following:

  ERROR: Failed to Upload File: /opt/emc/vproxy/runtime/tmp/vproxyd/
  plugin/search/e6c356a1-fbaf-4231-9f6f-a0166b74909a/<search 
  node>-e081fdea-3599-4a6c-abc4-1b5487cb9a32-e523a94c-2d01-5234-ab3c-
  7771cfab3c58-7f16bcbb72d7b49ea073356f0d7388ac08461827.db.zip to 
  https://<search node>:14251/upload, Error sending data chunk. Post 
  https://<search node>:14251/upload: x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "PPDM Root CA ID-d5ec56b8-69ec-4183-9c94-7c0230408765"

• Examine the REST engine logs in the Search Engine node (/opt/emc/search/logs/rest-engine/*.log), and look for certificate verification errors.
• Run a search either through the UI or through the API <PowerProtect Data Manager>/api/v2/file-instances and look for a certification verification error.

Examine the certificate files on each Search Engine node to investigate further. If necessary, regenerate the certificate files.

Verify certificates

Use this procedure to verify that certificates are valid and uncorrupted:

1. Verify that the rootca.pem file is the same in all the relevant nodes (Search Engine node, PowerProtect Data Manager, and VM Direct node).
   NOTE The rootca.pem file name is different on each node:

   • PowerProtect Data Manager—/etc/ssl/certificates/rootca/rootca.pem
   • Search Engine node—/var/lib/dellemc/vmboot/trust/thumbprint
   • VM Direct—/var/lib/dellemc/vmboot/trust/thumbprint
2. Run the following OpenSSL command to find out whether the root certificate file is corrupt or invalid: openssl verify <rootca.pem>

   Response:  

   /var/lib/dellemc/vmboot/trust/thumbprint: C = US, 
   O = DELL Corporation, 
   CN = PPDM Root CA ID-4c9de850-24ab-42ec-a9a7-6080849d0d24
   
   error 18 at 0 depth lookup:self signed certificate
   
   
   OK

   Ensure that the CN values match.

Certificate verification fails

If the certificate verification steps fail, you must re-create the certificates on the Search Engine node or VM Direct node:

1. Connect to the PowerProtect Data Manager console and change to the root user.
2. Use the Get command in the infranodemgmt utility to determine the Search Engine node FQDN.
3. Run /usr/local/brs/puppet/scripts/generate_certificates.sh -n -c -b <node FQDN>

   A properties file is created in the /root directory called <node FQDN>.properties.  

4. Open this file to determine the location of the generated certificates. They should be located in /etc/ssl/certificates/<node FQDN>.
5. Obtain the Search Engine node credentials. The PowerProtect Data Manager Security Configuration Guide provides instructions.
6. From a separate terminal, SSH into the Search Engine node.
7. Change directory to /var/lib/dellemc/vmboot/trust and move the key, cert, and thumbprint files over.
8. Copy the certificate files that were generated in PowerProtect Data Manager as follows:
   • rootca.pem to thumbprint
   • <search node FQDN>key.pem to key
   • <search node FQDN>.pem to cert
9. Paste the files to /var/lib/dellemc/vmboot/trust.
10. Set the permissions for the key, cert, and thumbprint files to 0644, and then set the ownership of these files to root:app.
11. Restart the REST engine service to pick up the new certificates: systemctl restart search-rest-engine.
12. Check the REST engine log file (/opt/emc/search/logs/rest-engine/rest-engine-daemon-<fqdn>.log) to verify that the service started successfully.

    Ensure that the following message appears:  

    A valid Root CA certificate of backup server was provided during deployment
    

Result: Backup with indexing executes successfully and the Search Engine is functional.

Search Engine cluster is full

If the Search Engine is full, you can deploy additional nodes by following the steps in Set up and manage indexing.

If the Search Engine runs out of space and you do not want to deploy an additional node, you have the following options:

• Disable the service
• Shorten the expiration time to remove indexes sooner
• Remove indexes manually

To disable the service, complete the following steps:

1. From the PowerProtect Data Manager UI, select Infrastructure > Search Engine.
2. Select the cluster, and then click Configure Cluster.
3. In the Configure Search Cluster dialog box, switch the Search Indexing button to turn it off, and then click Save.
   NOTE This setting applies to all indexes in all protection policies in the Search Cluster.

To shorten the expiration time to remove indexes sooner, complete the following steps:

1. From the PowerProtect Data Manager UI, select Infrastructure > Search Engine.
2. Select the cluster, and then click Configure Cluster.
3. In the Configure Search Cluster dialog box, modify the Search Index Expiration and click Save. A recommended formula to determine the expiration time is: Delete Index when Today = Backup-Date + Expiration Days + 1 day. That is, one day after the backup expires.
   NOTE This setting applies to all indexes in all protection policies in the Search Engine.

To remove indexes manually, complete the following steps:

1. Use SSH to log in to the Search Engine.
2. Create a snapshot of the cluster using the following format:

           {
               Command:  "APP_SNAPSHOT",
               Title:    "Initiate Index/Search Cluster Snapshot Process",
               AsyncCmd: false,
               Properties: {
                   "Name": {
                       Description: "Used to uniquely identify a particular snapshot",
                       Type:        STRING
                   },
                   "Action": {
                       Description: "Action to perform, 'Create', 'Delete', 'Restore' or 'Cancel' a Snapshot",
                       Type:        STRING
                   },
                   "NFSHost": {
                       Description: "NFS Host serving snapshot backup area.",
                       Type:        STRING
                   },
                   "NFSExport": {
                       Description: "NFS Export path to mount too.",
                       Type:        STRING
                   },
                   "NFSDirPath": {
                       Description: "NFS directory path to write too.",
                       Type:        STRING
                   }
               }
           }

   For example:  

   {
   	"Command": "APP_SNAPSHOT",
   	"Title": "",
   	"AsyncCmd": false,
   	"Properties": {
   		"Action": {
   			"Description": "",
   			"Required": false,
   			"Type": "string",
   			"IsArray": false,
   			"Value": "Create",
   			"Default": null
   		},
   		"Name": {
   			"Description": "",
   			"Required": false,
   			"Type": "string",
   			"IsArray": false,
   			"Value": "DataManager_Catalog_Cluster_snapshot_2019-10-16-12-57-16",
   			"Default": null
   		},
   		"NFSHost": {
   			"Value": "10.25.87.88"
   		},
   		"NFSExport": {
   			"Value": "/mnt/shared"
   		},
   		"NFSDirPath": {
   			"Value": ""
   		}
   	}
   } 

3. You can delete indexes by protection policy or by asset. If the JSON command is stored at /home/admin/remove-plc.json, run the command, ./searchmgmt -I /home/admin/remove-plc.json.
   • Use the following format to delete indexes by protection policy:

     {
                 "Command": "APP_REMOVE_ITEMS",
                 "AsyncCmd": false,
                 "Properties": {
                             "Action": {
                                         "Description": "Action to perform, 'AssetDelete', 'PLCDelete'",
                                         "Required": true,
                                         "Value": "PLCDelete",
                             }
                             "PLCID": {
                                         "Description": "PLC ID of item(s) to delete.",
                                         "Required": true,
                                         "Value": "7676d753-b57e-a572-6daf-33689933456d",
                             }
                 }
     }

   • Use the following format to delete indexes by asset type:

     {
                 "Command": "APP_REMOVE_ITEMS",
                 "AsyncCmd": false,
                 "Properties": {
                             "Action": {
                                         "Description": "Action to perform, 'AssetDelete', 'PLCDelete'",
                                         "Required": true,
                                         "Value": "AssetDelete",
                             },
                             "AssetID": {
                                         "Description": "Optional, Asset ID of item(s) to delete.",
                                         "Required": false,
                                         "Value": "503dd753-b57e-a572-6daf-44680033755f",
                             },
                             "PLCID": {
                                         "Description": "PLC ID of item(s) to delete.",
                                         "Required": true,
                                         "Value": "7676d753-b57e-a572-6daf-33689933456d",
                             }
                 }
     }

   NOTE

   • The time to complete the execution of these procedures depends on the number of backup copy asset indexes being deleted.
   • This procedure does not impact regular operation of the cluster.

Troubleshooting a locked Search Engine node

The PowerProtect Data Manager Security Configuration Guide provides information about Search Engine node user accounts and credentials, including password management policies. The password management policies for these accounts are set to lock the admin user account after three failed attempts within five minutes. If you try to access the node while the admin user account is locked, the amount of time that the account remains locked increases.

A Search Engine node might become locked for the following reasons:

• A user or program makes three failed attempts to SSH into the Search Engine node.
• Running monitoring software that tries to log in to the Search Engine node with the wrong admin credentials.
• Running penetration testing on the virtual machines in a vCenter server.

The Search Engine node admin user accounts enable PowerProtect Data Manager to perform operations on each node, such as obtaining the health status of the node. If the account is locked, the health status of the node is reported as "Failed." When one of the nodes in the cluster is in a failed state, the entire cluster becomes unavailable. As a result, the cluster is unable to perform any indexing or search operations.

Workaround

To work around this issue, reset the Search Engine node admin credentials. Before you reset the credentials, determine why the admin account is locked.

Obtain the Search Engine node root credentials. Then, reset the Search Engine node admin credentials. The PowerProtect Data Manager Security Configuration Guide provides instructions.