NVP-vProxy: Unable to install vCenter plug-in after replacing NetWorker certificate

The NetWorker VMware Protection integration is configured with the vProxy Appliance. Installation of the vCenter server plug-in is required to enable backup and recovery from the vCenter side.
When installing the plug-in, NetWorker reports the installation was successful, but vCenter reports a failure in deployment along with the following error in vsphere_client_virgo.log:

[YYYY-MM-DDTHH:MM:SSZ] [ERROR] -async-task-executor-pool-91  com.vmware.vise.extensionfw.plugins.impl.PluginStatusServiceImpl  DOWNLOAD_FAILED: Error downloading plugin package com.dell.emc.nw:19.4.0.95 from https://<NW-server>:9090/vcui/plugin.json. Reason: Download error. Make sure that the URL is reachable and the thumbprint is correct. com.vmware.vise.plugin.download.PluginDownloadException: javax.net.ssl.SSLHandshakeException: Server certificate chain is not trusted and thumbprint doesn't match
[YYYY-MM-DDTHH:MM:SSZ] [ERROR] -async-task-executor-pool-91  com.vmware.vise.vim.extension.VcExtensionManager                  Downloading plugin package: 'com.dell.emc.nw:19.4.0.95' registered in vCenter: '<vCenter server name>' has failed. java.util.concurrent.CompletionException: com.vmware.vise.plugin.download.PluginDownloadException: javax.net.ssl.SSLHandshakeException: Server certificate chain is not trusted and thumbprint doesn't match

The issue was observed for multiple environments where the NetWorker certificates were replaced post-installation with CA-certificates, for example:

• NetWorker: How to Import or Replace Certificate Authority Signed Certificates for "Authc" and "NWUI" (Linux)
• NetWorker: How to Import or Replace Certificate Authority Signed Certificates for "AUTHC" and "NWUI" (Windows)

As a workaround, deploy the Dell EMC NetWorker plug-in using vSphere Client Software Development Kit (SDK) CLI or register manually in vCenter MOB, updating the serverThumbprint record. A VMware Administrator must perform the actions outlined in this KB.

Work around 1 - Plugin registration using vSphere Client SDK:

1. Download the vSphere client SDK corresponding to the vCenter server version from the VMware website. You can keep the SDK in any Linux/Windows host which has the connectivity to NetWorker and vCenter server.

VMware vSphere Client SDK 8.0: vSphere Client SDK - 8.0 

NOTE: See the Broadcom Developer Portal  for other VMware version SDK downloads.

2. Extract the SDK package and go to path: html-client-sdk/vCenter plug-in registration/prebuilt/

• This path contains extension-registration.sh and extension-registration.bat that can be run remotely on the host where the SDK is downloaded depending on the operating system.

3.  Run the following command on vCenter server appliance/Linux NetWorker server and take a note of the generated NetWorker server certificate thumbprint:

keytool -printcert -sslserver <NetWorker Server IP/FQDN>:9090 -rfc | openssl x509 -fingerprint -noout

4. Run the following command on the vCenter server appliance/Linux NetWorker server and take a note of the generated vCenter server certificate thumbprint:

keytool -printcert -sslserver <vCenter Server IP/FQDN>:443 -rfc | openssl x509 -fingerprint -noout

5. Run the script using the following syntax, replacing environment-specific values.

• On Windows:

extension-registration.bat -action registerPlugin -remote -url https://vCenter-server-name-or-IP/sdk -username administrator@vsphere.local -password vCenter-password -key com.dell.emc.nw -version NetWorker-Version-see-below-note -pluginUrl https://NW-server-name-o-IP:9090/vcui/plugin.json -serverThumbprint NetWorker-thumbprint-from-step-3 -vct vCenter-thumbprint-from-step-4 -c "Dell EMC" -n "NW Data Protection" -s "VCUI - vSphere Data Protection NetWorker"

• On Linux:
• Make the extension-registration.sh executable by running the command:

chmod +x extension-registration.sh

• Run the script:

./extension-registration.sh -action registerPlugin -remote -url https://vCenter-server-name-or-IP/sdk -username administrator@vsphere.local -password vCenter-password -key com.dell.emc.nw -version NetWorker-Version-see-below-note -pluginUrl https://NW-server-name-o-IP:9090/vcui/plugin.json -serverThumbprint NetWorker-thumbprint-from-step-3 -vct vCenter-thumbprint-from-step-4 -c "Dell EMC" -n "NW Data Protection" -s "VCUI - vSphere Data Protection NetWorker"

6. Logout and log in to the vSphere web interface.
7. Go to Plugin and enter NetWorker NMC credentials and login to perform backup and restore using the VCUI plug-in. Once you initiate backups or restore, the com.emc.networker.backup, and com.emc.networker.recover extensions should be populated in vCenter MOB.

Workaround 2 - Manual extension registration from MOB:

1. Obtain the NetWorker certificate thumbprint using the following command on VCSA:

keytool -printcert -sslserver <NetWorker Server IP/FQDN>:9090 -rfc | openssl x509 -fingerprint -noout

1. Log in to the MOB extension manager: https://VCENTER_ADDRESS/mob/?moid=ExtensionManager
2. Select Register Extension.
3. Use the attached com.dell.emc.nw extension configuration file as the extension value.
   • NOTE: You must modify the following parameters/tag values in com.dell.emc.nw extension value to reflect your environment configuration:
     • Version -- your current NW version and build number, for example: If using NetWorker 19.12.0.2.build.57, the version should be 19.12.0.57.
     • URL -- https://NW server name or IP:9090/vcui/plugin.json
     • serverThumbprint -- from step 1
     • lastHeartbeatTime -- current date and time in the format YYYY-MM-DDTHH:MM:SSZ
4. Check the status of deployment on the vCenter server’s "Client Plug-in" tab and recent tasks. 
5. Logout and log in to the vSphere web interface.

Workaround 3 - Manual extension registration from the vCenter Server command-line:

1. Open a root shell on the vCenter server.
2. Initiate the NetWorker plugin installation from the vSphere using REST API command-line:

curl -kvi \
  -H 'Content-Type: application/json' \
  --user 'Administrator:PASSWORD' \
  -d '{"pluginType":"VC",
       "httpsPort":"443",
       "nwUserId":"administrator",
       "nwPassword":"PASSWORD"}' \
  'https://IP_ADDRESS:9090/nwrestapi/v3/global/vmware/vcenters/VCENTER_NAME/plugins'

Replace:

• • PASSWORD with the password for the NetWorker Administrator account.
  • IP_ADDRESS with the NetWorker Server IP address that the vCenter can communicate with.
  • VCENTER_NAME with the name of the vCenter as it is registered in NetWorker.

root@vcsa [ ~ ]# curl -kvi \
>   -H 'Content-Type: application/json' \
>   --user 'Administrator:!Password1' \
>   -d '{"pluginType":"VC",
>        "httpsPort":"443",
>        "nwUserId":"administrator",
>        "nwPassword":"!Password1"}' \
>   'https://192.168.9.150:9090/nwrestapi/v3/global/vmware/vcenters/vcsa.amer.lan/plugins'
...
...
* Server auth using Basic with user 'Administrator'
> POST /nwrestapi/v3/global/vmware/vcenters/vcsa.amer.lan/plugins HTTP/1.1
> Host: 192.168.9.150:9090
...
* upload completely sent off: 114 bytes
< HTTP/1.1 200
HTTP/1.1 200
...
{
  "resultCode" : "Success",
  "data" : "Registration Success",
  "debug" : "",
  "responseString" : null
* Connection #0 to host 192.168.9.150 left intact

The NetWorker plugin is listed in the vCenter server's extension manager: https://VCENTER_ADDRESS/mob/?moid=ExtensionManager&doPath=extensionList%5b%22com.dell.emc.nw%22%5d.server