VxRail: Unable to Log In to vCenter Due to Expired Certificates

Table of Contents

Detailed Article

Symptoms

Cause

Resolution

Additional Info

Affected Products

Provide Feedback

Summary: VxRail 4.5 and 4.7: Unable to log in to vCenter due to expired certificates. Certificates must be re-issued. VxRail 7.0.480 or later: A Warning shows for certificate expires in less than 60 days, recommend renewing the certificate in advance. ...

This article applies to This article does not apply to

Check out resources for

Symptoms

Scenario 1: The vCenter certificate is already expired. (For all VxRail versions)

Unable to log in to vCenter UI.
Any log-in attempt when the Web UI is available fails even with correct credentials.
Restart of vCenter Server Appliance (VCSA) services fails.
Restart of services does not bring up all services.

Errors observed:

/var/log/vmware/vpxd-svcs/vpxd-svcs.log:
2020-06-03T09:31:04.523Z [pool-8-thread-1  INFO  com.vmware.identity.token.impl.X509TrustChainKeySelector  opId=905f6864-c067-4db6-828c-1d59c4b43bf8] Failed to find trusted path to signing certificate <CN=ssoserverSign>
sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
        at sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141)

Scenario 2: The vCenter certificate expires in less than 60 days. (For VxRail 7.0.480 and above versions)

Log in to vCenter UI is completed but VxRail 7.0.480 and later versions show a Warning in VxRail Cluster > Configure > VxRail > Certificate > All Trust Store Certificates page stating that the certificate expires in less than 60 days.

Cause

vCenter certificates are expired or expire soon.
VxRail versions which were initially built prior to 4.7, may have certificates issued with a lifespan of two years from the date of installation. At the time of writing of this article, a VxRail build on 4.7.410 has all certificates with a 10-year lifespan.

Minor version upgrades do not touch the certificates!
For a VxRail which was initially built on 4.5.210 and later versions, the certificates have a two-year validity period. Check the VMware article for VMware Security Token Service (STS) Checking Expiration of STS Certificate on vCenter Servers (79248) to confirm the detailed description.

Use the view certificate in the browser of the log-in page of the VCSA to confirm the certificate has expired. Or list the certificates in the CLI of the Platform Services Controller (PSC) (VCSA). See commands from VMware article Signing certificate is not valid" error in VCSA 6.5.x,6.7.x or vCenter Server 7.0.x, 8.0.x. (76719)

for i in $(/usr/lib/vmware-vmafd/bin/vecs-cli store list); do echo STORE $i; /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store $i --text | egrep "Alias|Not After"; done

Resolution

For Scenario 1, when the vCenter certificate is already expired, follow the below procedure to generate new self-signed certificates on PSC and VCSA.

Note: This procedure is intended for single PSC or VCSA VMs which are maintained through VxRail LifeCycle Manager (LCM). For HA, ELM, or Customer deployed VCSAs, open a VMware ticket!

Note: Take OFFLINE snapshots of VxRail Manager (VRM), PSC, and VCSA!

Note: Check if the snapshot creating process has finished without errors! Do NOT continue without valid snapshots!

Note: If issues are encountered, do not retry without reverting to snapshots!

Fix PSC:
Reset all Certificates (This fails but that is expected.)
- Start Certificate Manager:
```
/usr/lib/vmware-vmca/bin/certificate-manager
```
- Select Option 8 > Reset all Certificates
  - Confirm
```
"Do you wish to generate all certificates using configuration file : Option[Y/N] ?"
```
  - Enter Credentials
  - Enter Values
    - Leave "IPAddress" field empty
    - Enter Hostname as Fully Qualified Domain Name (FQDN) of PSC
    - VMware Certificate Authority (VMCA) Name field is the name of a new root CA being created, for example, VxRail CA.
  - Confirm
```
"Continue operation : Option[Y/N] ?"
```
  - Confirm
```
"Continue operation : Option[Y/N] ?"
```
    - This operation fails with:
```
Get site nameCompleted [Reset Machine SSL Cert...]
g3node-site
Lookup all services
Get service xxxxxx-site:xxxxxxx-d202-4d8f-9282-xxxxxx317b8f
Update service xxxxxx-site:xxxxxxxx-d202-4d8f-9282-xxxxxx317b8f; spec: /tmp/svcspec_a1hipoqq

Status : 0% Completed [Reset operation failed]

please see /var/log/vmware/vmcad/certificate-manager.log for more information.
root@xxxxc [ ~ ]#
```
  - Fix the STS issue
    Download and run the script from VMware article "Signing certificate is not valid" error in VCSA 6.5.x,6.7.x or vCenter Server 7.0.x, 8.0.x. (76719)
    - Stop Services
```
service-control --all --stop
```
    - Start Services (This fails but that is expected)
```
service-control --all --start
```
    - Wait for the process to time out or stop it when it gets to the "vmware-vmon" service
```
/usr/lib/vmware-vmca/bin/certificate-manager
```
    - Select Option 6 > "Replace Solution user certificates with VMCA certificates"
    - Confirm
```
"Do you wish to generate all certificates using configuration file : Option[Y/N] ?"
```
    - Enter Credentials
    - Deny (enter "N") for reconfigure as all options were configured above
```
"certool.cfg file exists, Do you wish to reconfigure : Option[Y/N] ?"
```
    - Confirm
```
"You are going to regenerate Solution User Certificates using VMCA, Continue operation : Option[Y/N] ?"
```
    - Wait until the procedure exits. This procedure:
      - Generates all certificates
      - Stops the services
      - Starts the services
    - Confirm if all Services are running
```
service-control --all --status
```
  - Fix Certificates on VCSA
    Stop and start all services. This MUST be done AFTER all PSC services are running!
    - Stop
```
service-control --all --stop
```
    - Start
```
service-control --all --start
```
    - Wait for the process to time out or stop it when it gets to the vmware-vmon service
```
/usr/lib/vmware-vmca/bin/certificate-manager
```
    - Select Option 8 > Reset all Certificates
    - Start Certificate Manager
    - Confirm
```
"Do you wish to generate all certificates using configuration file : Option[Y/N] ?"
```
    - Enter Credentials
    - Enter PSC IP
    - Enter Values
      - Leave IPAddress field empty
      - Enter Hostname as FQDN of VCSA
      - VMCA Name field is the name of the new root CA being created, for example, VxRail CA.
    - Confirm
```
"Continue operation : Option[Y/N] ?"
```
    - Confirm
```
"Continue operation : Option[Y/N] ?"
```
    - Wait until all certificates are generated and a successful completion message appears
```
"Reset status : 100% Completed [Reset completed successfully]"
```
    - Check that all services are running
```
service-control --all --status
```
    - Access vCenter UI
    - Access by Domain Name System (DNS) entry fails in Chrome due to HTTP Strict Transport Security (HSTS). Open the VCSA IP or use another supported browser such as FireFox.

For Scenario 2, when the vCenter certificate expires in less than 60 days, follow the below procedure to renew the certificate in advance to avoid VxRail manager disconnect from vCenter.

Log in to the vCenter over SSH as the root user

Restart Services

Run Stop
```
"service-control --stop --all"
```
Run Start
```
"service-control --start --all"
```

Reset all Certificates
- Run:
```
"/usr/lib/vmware-vmca/bin/certificate-manager"
```
- Select Option 8 > Reset all Certificates
- Enter vSphere username and password
- Input the Certificate Properties
- Confirm the operation and then the vCenter root or machine Certificates are renewed
Follow the article Dell VxRail: How to manually import vCenter SSL certificate on VxRail Manager to import the updated vCenter and CA certificates into the VxRail Manager trust store.

Additional Information

ALWAYS take snapshots of System VMs (PSC, VCSA, and VRM) before following this article.
This procedure is intended for PSC VCSA VMs which are maintained through VxRail LCM.
NOTE! Some third-party products must be re-registered or add the new VMCA Root CA to be trusted (product specific - check product documentation)
- This as communication is broken due to Root or VCSA certificate change
If a user has certificates from their own infrastructure, they can replace them now.
After fix for VxRail version 4.7.100 and later, follow the KB article Dell VxRail: How to manually import vCenter SSL certificate on VxRail Manager to import a new root certificate into VRM (plug-in does not work).

Affected Products

VxRail Appliance Family, VxRail Appliance Series

VxRail: Unable to Log In to vCenter Due to Expired Certificates