Data Domain: After upgrading to DDOS or DDMC 6.2.1.90, 7.2.0.95, or 7.7.2.x or later, UI cannot be accessed

A user upgrades either DDOS or DDMC to version 7.7.2.x, 7.7.3.x, 7.8.x, or 7.9.x and after the upgrade is complete, discover it is not possible to access the UI. Restarting the HTTP or HTTPS services from the command line does not make it work either. When using a browser to access the UI, the following error page shows: 

Service Not Available
The GUI Service is temporarily unavailable. Refresh browser to try again. If the problem persists, contact Dell EMC support for assistance.

The UI backend is running on a Java-based application. UI backend is not running. If the DD was upgraded to any of the mentioned releases and the UI is failing, if the DD has a trust relationship with other DDs and any of them have an incorrect certificate, it may be the cause of why the DD UI is not starting up. A stricter check in the newer bundled JDK fails to pass for some of the trusted DD hosts.

For help with checking if a DLm is attached, see Dell KB article 207823: 
Data Domain + DLM: After upgrading to DDOS or DDMC 6.2.1.90, 7.2.0.95, or 7.7.2.x or later, UI cannot be accessed.
https://www.dell.com/support/kbdoc/en-us/000207823 Data Domain + DLM: After upgrading to DDOS or DDMC 6.2.1.90, 7.2.0.95, or 7.7.2.x or later, UI cannot be accessed

 
It is necessary to ascertain if this is exactly the issue being faced. For this to be the case, all the conditions below must hold true:

1. DD where UI is not starting up is experiencing the UI issues only since upgrading to either:
   • DDOS 7.7.2.x
   • DDOS 7.7.3.x
   • DDOS 7.8.x
   • DDOS 7.9.x
2. This DD has a trust relationship with other DDs, one or more of which has a DDOS 4.7 - 5.4 version in the past.
3. This DD has a particular set of logs for the failure to start the UI backend.

Item one is self-explanatory.

To determine if item two applies, get the list of trusted hosts in the DD:

# adminaccess trust show

Subject                  Type         Valid From                 Valid Until                Fingerprint
----------------------   ----------   ------------------------   ------------------------   -----------------------------------------------------------
dd-upgraded-7.7.2.x      trusted-ca   Tue Jul  6 15:36:35 2021   Mon Jul  5 15:36:35 2027   BA:59:F0:6E:93:47:02:7C:6C:C1:39:71:46:6D:1F:2B:81:09:E9:46
dd-trusted-1             trusted-ca   Mon Feb  7 10:26:48 2011   Thu Jan 30 10:26:48 2042   7B:96:6B:06:D9:A4:6A:B1:A0:3A:4C:E7:12:28:07:AD:29:F3:8E:F2
dd-trusted-2             trusted-ca   Mon Aug 21 10:18:52 2017   Thu Aug 13 10:18:52 2048   16:BC:09:02:F5:39:CB:EC:C2:A8:9E:5D:21:90:19:38:2B:36:47:EA
----------------------   ----------   ------------------------   ------------------------   -----------------------------------------------------------

In this case, "dd-upgraded-7.7.2.x" is the one that has the UI failing and the other two are the trusted DDs (because they have replication contexts set with the local DD). Unless regenerated later, a DD certificate used for trust is "valid from" the install date. In the output above, the most likely candidate for running an old DDOS 4.7 - 5.4 release is "dd-trusted-1."

Log in to that DD and check it (or look for the "System Upgrade History" section in ASUPs):

# system upgrade history
Version           Partition   State     Time                Package
                  Number
---------------   ---------   -------   -----------------   ------------
5.1.0.5-269447    1           INSTALL   12/29/11 01:02:59
5.1.0.9-282511    2           UPGRADE   02/07/12 13:31:42   ddr
5.4.5.0-477080    1           UPGRADE   05/04/15 10:22:42   ddr
5.5.4.10-536339   2           UPGRADE   04/17/18 08:51:19   ddr
5.7.5.0-569395    1           UPGRADE   04/17/18 09:41:17   ddr
---------------   ---------   -------   -----------------   ------------

In this output, "dd-trusted-1" is a match for item two, and is likely to have an incorrect certificate causing the "dd-upgraded-7.7.2.x" DD host to fail loading the UI through their trust relationship.

To confirm item three (if the DD UI failure logs are for this specific issue), run the following command to open the "em.info" log file:

# log view debug/sm/em.info

Search (use a forward slash) for these logs:

05 Jul 2022 13:57:38,737 INFO  [main] Reloading the certificate stores for the system
05 Jul 2022 13:57:38,899 ERROR [main] Exception during command execution: java.security.cert.CertificateException - Invalid X.509 Certificate version., will retry,  Attempt# 1
05 Jul 2022 13:57:39,001 ERROR [main] Exception during command execution: java.security.cert.CertificateException - Invalid X.509 Certificate version., will retry,  Attempt# 2
05 Jul 2022 13:57:39,103 ERROR [main] Exception during command execution: java.security.cert.CertificateException - Invalid X.509 Certificate version., will retry,  Attempt# 3

The most immediate workaround is to remove trust with any DD host which had run DDOS 4.7 - 5.4 in the past, so that in the case it has a wrong certificate created, it does not cause the local DD UI to fail loading up anymore. Continuing with the example, let us say dd-trusted-1 is the only one of the trusted DD that ever runs DDOS 5.7 - 5.4. To remove mutual trust (this DD trusting dd-trusted-1, and the opposite way), from the DD with the UI issue do the following steps.
 

# adminaccess trust del host dd-trusted-1 type mutual

Removing trust causes replication set up with the now untrusted DD to fail until trust has been reestablished. If removing trust fails because the remote DD is no longer reachable or does not even exist, repeat the command without the "mutual" option, to only remove trust locally, as that is sufficient to get the user interface to work again.

For cases in which the trusted DD is still in use and we must keep the trust relationship, a new CA certificate has to be created:

1. Generate a new self-signed CA certificate on the offending system, from the CLI:

# adminaccess certificate generate self-signed-cert regenerate-ca

 

2. Fix the trust that this DD system maintains with every other DD or DDMC system (mutual "adminaccess trust del" followed by "adminaccess trust add" from each device)

3. Finally, cycle the web service in any of the affected devices:

   # adminaccess disable https
   # adminaccess enable https

If after doing the above the DD user interface does not work yet, contact DELL Data Domain Support.

To reestablish trust, you must first ensure the DD CA certificate in "dd-trusted-1" is re-created without any errors. From this host command line, do so by running the command below, and follow the prompts if any: 

# adminaccess certificate generate self-signed-cert regenerate-ca

If this DD has trust established with other DDs or replication, you must delete and then readd trust for all such peers.

Finally, to readd trust in between "dd-upgraded-7.7.2.x" and "dd-trusted-1", either:

• Run "adminaccess trust add host dd-trusted-1 type mutual" from "dd-upgraded-7.7.2.x", or 
• Run "adminaccess trust add host dd-upgraded-7.7.2.x type mutual" from "dd-trusted-1"