VxRail: Unable to Log In to vCenter Due to Expired Certificates

Scenario 1: The vCenter certificate is already expired. (For all VxRail versions)

• Unable to log in to vCenter UI.
• Any log-in attempt when the Web UI is available fails even with correct credentials.
  
• Restart of vCenter Server Appliance (VCSA) services fails.
• Restart of services does not bring up all services.

/var/log/vmware/vpxd-svcs/vpxd-svcs.log:
2020-06-03T09:31:04.523Z [pool-8-thread-1  INFO  com.vmware.identity.token.impl.X509TrustChainKeySelector  opId=905f6864-c067-4db6-828c-1d59c4b43bf8] Failed to find trusted path to signing certificate <CN=ssoserverSign>
sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
        at sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141)

Scenario 2: The vCenter certificate expires in less than 60 days. (For VxRail 7.0.480 and above versions)

• Log in to vCenter UI is completed but VxRail 7.0.480 and later versions show a Warning in VxRail Cluster > Configure > VxRail > Certificate > All Trust Store Certificates page stating that the certificate expires in less than 60 days.
  

vCenter certificates are expired or expire soon.
VxRail versions which were initially built prior to 4.7, may have certificates issued with a lifespan of two years from the date of installation. At the time of writing of this article, a VxRail build on 4.7.410 has all certificates with a 10-year lifespan.

Minor version upgrades do not touch the certificates!
For a VxRail which was initially built on 4.5.210 and later versions, the certificates have a two-year validity period. Check the VMware article for VMware Security Token Service (STS) Checking Expiration of STS Certificate on vCenter Servers (79248) to confirm the detailed description.

Use the view the certificate in the browser of the log-in page of the VCSA to confirm the certificate has expired. Or list the certificates in the CLI of the Platform Services Controller (PSC) (VCSA). See commands from VMware article Signing certificate is not valid" error in VCSA 6.5.x,6.7.x or vCenter Server 7.0.x, 8.0.x. (76719)

for i in $(/usr/lib/vmware-vmafd/bin/vecs-cli store list); do echo STORE $i; /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store $i --text | egrep "Alias|Not After"; done

For Scenario 1, when the vCenter certificate is already expired, follow the below procedure to generate new self-signed certificates on PSC and VCSA.

1. Fix PSC:
   Reset all Certificates (This fails but that is expected.)

   • Start Certificate Manager:

     /usr/lib/vmware-vmca/bin/certificate-manager

   • Select Option 8 > Reset all Certificates
     • Confirm

       "Do you wish to generate all certificates using configuration file : Option[Y/N] ?"

     • Enter Credentials
       
     • Enter Values
       • Leave the "IPAddress" field empty
       • Enter Hostname as Fully Qualified Domain Name (FQDN) of PSC
       • VMware Certificate Authority (VMCA) Name field is the name of a new root CA being created, for example, VxRail CA.
     • Confirm

       "Continue operation : Option[Y/N] ?"

     • Confirm

       "Continue operation : Option[Y/N] ?"

       • This operation fails with:

         Get site nameCompleted [Reset Machine SSL Cert...]
         g3node-site
         Lookup all services
         Get service xxxxxx-site:xxxxxxx-d202-4d8f-9282-xxxxxx317b8f
         Update service xxxxxx-site:xxxxxxxx-d202-4d8f-9282-xxxxxx317b8f; spec: /tmp/svcspec_a1hipoqq
         
         Status : 0% Completed [Reset operation failed]
         
         please see /var/log/vmware/vmcad/certificate-manager.log for more information.
         root@xxxxc [ ~ ]#

         

     • Fix the STS issue
       Download and run the script from VMware article "Signing certificate is not valid" error in VCSA 6.5.x,6.7.x or vCenter Server 7.0.x, 8.0.x. (76719)
       

       • Stop Services

         service-control --all --stop

       • Start Services (This fails but that is expected)

         service-control --all --start

         
       • Wait for the process to time out or stop it when it gets to the "vmware-vmon" service

         /usr/lib/vmware-vmca/bin/certificate-manager

       • Select Option 6 > "Replace Solution user certificates with VMCA certificates"
       • Confirm

         "Do you wish to generate all certificates using configuration file : Option[Y/N] ?"

       • Enter Credentials
       • Deny (enter "N") for reconfigure as all options were configured above

         "certool.cfg file exists, Do you wish to reconfigure : Option[Y/N] ?"

       • Confirm

         "You are going to regenerate Solution User Certificates using VMCA, Continue operation : Option[Y/N] ?"

       • Wait until the procedure exits. This procedure:
         • Generates all certificates
         • Stops the services
         • Starts the services
           
       • Confirm if all Services are running

         service-control --all --status

         

     • Fix Certificates on VCSA
       Stop and start all services. This MUST be done AFTER all PSC services are running!

       • Stop

         service-control --all --stop

       • Start

         service-control --all --start

         
       • Wait for the process to time out or stop it when it gets to the vmware-vmon service

         /usr/lib/vmware-vmca/bin/certificate-manager

       • Select Option 8 > Reset all Certificates
       • Start Certificate Manager
       • Confirm

         "Do you wish to generate all certificates using configuration file : Option[Y/N] ?"

       • Enter Credentials
         
       • Enter PSC IP
       • Enter Values
         • Leave IPAddress field empty
         • Enter Hostname as FQDN of VCSA
         • The VMCA Name field is the name of the new root CA being created, for example, VxRail CA.
       • Confirm

         "Continue operation : Option[Y/N] ?"

       • Confirm

         "Continue operation : Option[Y/N] ?"

         
       • Wait until all certificates are generated and a successful completion message appears

         "Reset status : 100% Completed [Reset completed successfully]"

         
          
         
       • Check that all services are running

         service-control --all --status

         
       • Access vCenter UI
       • Access by Domain Name System (DNS) entry fails in Chrome due to HTTP Strict Transport Security (HSTS). Open the VCSA IP or use another supported browser such as FireFox.
         
         

For Scenario 2, when the vCenter certificate expires in less than 60 days, follow the below procedure to renew the certificate in advance to avoid VxRail manager disconnect from vCenter.

1. Log in to the vCenter over SSH as the root user

2. Restart Services

   • Run Stop

     "service-control --stop --all"

   • Run Start

     "service-control --start --all"

3. Reset all Certificates

   • Run:

     "/usr/lib/vmware-vmca/bin/certificate-manager"

   • Select Option 8 > Reset all Certificates
     
   • Enter vSphere username and password
     
   • Input the Certificate Properties
     
   • Confirm the operation and then the vCenter root or machine Certificates are renewed
     

4. Follow the article Dell VxRail: How to manually import vCenter SSL certificate on VxRail Manager to import the updated vCenter and CA certificates into the VxRail Manager trust store.

• ALWAYS take snapshots of System VMs (PSC, VCSA, and VRM) before following this article.
• This procedure is intended for PSC VCSA VMs which are maintained through VxRail LCM.

• If a user has certificates from their own infrastructure, they can replace them now.
• After fix for VxRail version 4.7.100 and later, follow the KB article Dell VxRail: How to manually import vCenter SSL certificate on VxRail Manager to import a new root certificate into VRM (plug-in does not work).