Data Domain: How to handle upcoming Microsoft Azure Storage Transport Layer Security certificate changes for systems that are configured with Cloud Tier or systems that are deployed on the Azure cloud platform

Should the Azure Storage certificate change occur (starting in July 2022) before the new certificates are added as trusted for cloud, the cloud unit goes into a disconnect state for a Data Domain system that is configured with Cloud Tier in Azure:

# alert show current
Id      Post Time                  Severity   Class   Object                     Message
-----   ------------------------   --------   -----   ------------------------   -------------------------------------------------------------------------
m0-76   Mon Apr 19 15:34:03 2021   CRITICAL   Cloud   CloudUnit=azure-unit   EVT-CLOUD-00001: Unable to access provider for cloud unit azure-unit.
-----   ------------------------   --------   -----   ------------------------   -------------------------------------------------------------------------
There is 1 active alert.
# cloud unit list
Name             Profile     Status
--------------   ---------   ------------
azure-unit        azure      Disconnected
--------------   ---------   ------------

If Data Domain Virtual Edition (DDVE) is deployed on Azure with Active Tier on Object Storage (ATOS), the file system is disabled with the following alerts messages:

Alert History
-------------
Id      Post Time                  Clear Time                 Severity   Class               Object   Message
-----   ------------------------   ------------------------   --------   -----------------   ------   --------------------------------------------------------------------------------------
m0-26   Tue Apr  6 13:58:41 2021   Tue Apr  6 13:59:03 2021   ERROR      Filesystem                 EVT-FILESYS-00008: Filesystem has encountered an error and is restarting.
m0-27   Tue Apr  6 14:19:59 2021   Tue Apr  6 14:20:03 2021   ALERT      Filesystem                 EVT-FILESYS-00002: Problem is preventing filesystem from

Microsoft Azure Storage services were updated to use TLS certificates from Certificate Authorities (CAs) that chain up to the DigiCert Global G2 root. However, in the meantime the current certificates (issued by the Baltimore Cyber-Trust root) continue being used.

This change is starting from July 2022 and expected to be complete by the end of October 2022. We recommend installing a new certificate before June 30, 2022 and DO NOT delete current certificate.

To access Blob containers (for either Data Domain Cloud Tier or DDVE ATOS), Data Domain systems require the new DigiCert Global G2 root CA certificate instead of the current Baltimore Cyber-Trust root CA certificate as trusted for cloud.

For more details about the topic, see the following official Azure security Blob:
Azure Storage TLS: Critical changes are almost here! (…and why you should care) - Microsoft Tech Community.

To make the transition to the new Azure Storage security certificates as smooth as possible when the changes start being pushed in July 2022, it is necessary to keep the trusted "Baltimore Cyber-Trust root CA" certificate as trusted for cloud in Data Domain, and add the new "DigiCert Global G2 Root CA" certificate also as trusted for cloud, rather than replacing one with another.

Keeping both certificates incurs no issue and allows the Data Domain DDVE system to trust the connections to Azure Storage no matter the certificate being presented issued by either of the CAs, and so avoiding any downtime as the new certificate is presented to the Data Domain/ DDVE system for the first time at some point starting in July 2022.

The following steps are applicable to support DigiCert Global G2 root certificate for Data Domain systems that are configured either with Cloud Tier or DDVE deployed on Azure Cloud platform with ATOS. It is also recommended to add DigiCert Global G3 root certificate and Microsoft ECC or RSA Root Certificate Authority Certificate to avoid future disruptions.

Confirm that Data Domain DDVE system has "Baltimore Cyber-Trust Root" for cloud application as per the following example:

sysadmin@dd01# adminaccess certificate show
Subject                     Type            Application   Valid From                 Valid Until                Fingerprint
-------------------------   -------------   -----------   ------------------------   ------------------------   -----------------------------------------------------------
dd01.example.com             host            https         Tue Mar 26 10:38:34 2019   Wed Jan 31 10:48:38 2024   30:78:FE:93:DF:2F:9D:B5:08:D7:EC:5E:9E:89:E2:BD:16:13:E1:BA
dd01.example.com             ca              trusted-ca    Wed Mar 27 17:38:34 2019   Wed Jan 31 10:16:38 2024   CB:9D:64:39:56:48:FB:58:C6:93:40:FB:29:91:56:9A:BD:08:7A:C8
Baltimore CyberTrust Root    imported-ca     cloud         Fri May 12 11:46:00 2000   Mon May 12 16:59:00 2025   D4:DE:20:D0:5E:66:FC:53:FE:1A:50:88:2C:78:DB:28:52:CA:E4:74
-------------------------   -------------   -----------   ------------------------   ------------------------   -----------------------------------------------------------

OPTION 1: Instructions for installing Certificate using Microsoft Windows workstation
For a Demo, click the below video:

Watch on YouTube.

1. Create a folder on your local workstation.
   For example:
   C:\MS-AZ-certificates

2. Download DigiCert Global Root G2/G3 certificates from the following page:
   DigiCert Root Certificates - Download & Test | DigiCert.com

   Right click on Download PEM and save the link as:
   DigiCertGlobalRootG2.crt.pem
   SHA1 Fingerprint: DF:3C:24:F9:BF:D6:66:76:1B:26:80:73:FE:06:D1:CC:8D:4F:82:A4

   DigiCertGlobalRootG3.crt.pem
   SHA1 Fingerprint: 7E:04:DE:89:6A:3E:66:6D:00:E6:87:D3:3F:FA:D9:3B:E8:3D:34:9E

3. Download Microsoft RSA and ECC certificate.

   Right click on Download PEM and save the link as:
   Microsoft RSA Root Certificate Authority 2017
   (Thumbprint: 73a5e64a3bff8316ff0edccc618a906e4eae4d74)

   Microsoft ECC Root Certificate Authority 2017
   (Thumbprint: 999a64c37ff47d9fab95f14769891460eec4c3c5)

4. Go to the command line and run the steps below. These are to run from a Windows host, however, similar commands may run from a Linux host.

C:\MS-AZ-certificates>dir
 Volume in drive C is OS
 Volume Serial Number is E4AE-2A04

 Directory of C:\MS-AZ-certificates

15/10/2021  09:30 AM    <DIR>          .
15/10/2021  09:30 AM    <DIR>          ..
15/10/2021  09:29 AM             1,294 DigiCertGlobalRootG2.crt.pem
15/10/2021  09:29 AM               839 DigiCertGlobalRootG3.crt.pem
15/10/2021  09:30 AM               605 Microsoft ECC Root Certificate Authority 2017.crt
15/10/2021  09:30 AM             1,452 Microsoft RSA Root Certificate Authority 2017.crt
               4 File(s)          4,190 bytes

5. Rename DigiCertGlobalRootG2 files as follows with .pem.

C:\MS-AZ-certificates>rename DigiCertGlobalRootG2.crt.pem DigiCertGlobalRootG2.pem

C:\MS-AZ-certificates>rename DigiCertGlobalRootG3.crt.pem DigiCertGlobalRootG3.pem

6. Convert Microsoft ECC RSA Root Certificate Authority Certificate from crt to pem format as follows:

C:\MS-AZ-certificates>certutil -encode "Microsoft ECC Root Certificate Authority 2017.crt" ms-ecc-root.pem
Input Length = 605
Output Length = 890
CertUtil: -encode command completed successfully.

C:\MS-AZ-certificates>certutil -encode "Microsoft RSA Root Certificate Authority 2017.crt" ms-rsa-root.pem
Input Length = 1452
Output Length = 2054
CertUtil: -encode command completed successfully.

C:\MS-AZ-certificates>dir
 Volume in drive C is OS
 Volume Serial Number is E4AE-2A04

 Directory of C:\MS-AZ-certificates

15/10/2021  10:25 AM    <DIR>          .
15/10/2021  10:25 AM    <DIR>          ..
15/10/2021  09:29 AM             1,294 DigiCertGlobalRootG2.pem
15/10/2021  09:29 AM               839 DigiCertGlobalRootG3.pem
15/10/2021  09:46 AM               605 Microsoft ECC Root Certificate Authority 2017.crt
15/10/2021  09:45 AM             1,452 Microsoft RSA Root Certificate Authority 2017.crt
15/10/2021  10:25 AM               890 ms-ecc-root.pem
15/10/2021  10:25 AM             2,054 ms-rsa-root.pem
               6 File(s)          7,134 bytes

7. Import all PEM-formatted certificate files from the folder, by using the PowerProtect DD System Manager UI.

   • For Data domain system configured with Cloud Tier:
     Data Management -> File System -> Cloud Units -> Manage Certificates -> Add.
     
     
     
     Repeat the Above steps for the remaining three certificates.
   • For Data Domain system running on Azure Platform with ATOS:
     Data Management -> File System -> Summary -> Modify Object Store -> CERTIFICATE -> Add.
     
     
     Repeat the Above steps for the remaining three certificates.

      

     Note: Do not add new Certification Authority certificates Under Access management.

OPTION 2: Instructions for installing Certificate using Linux workstation

1. Download the following two DigiCert Certificates in .pem format.

   https://cacerts.digicert.com/DigiCertGlobalRootG2.crt.pem
   SHA1 Fingerprint: DF:3C:24:F9:BF:D6:66:76:1B:26:80:73:FE:06:D1:CC:8D:4F:82:A4

   https://cacerts.digicert.com/DigiCertGlobalRootG3.crt.pem
   SHA1 Fingerprint: 7E:04:DE:89:6A:3E:66:6D:00:E6:87:D3:3F:FA:D9:3B:E8:3D:34:9E

2. Download the following two Root Certificates are in DER CRT format.

   Microsoft RSA Root Certificate Authority 2017
   (Thumbprint: 73a5e64a3bff8316ff0edccc618a906e4eae4d74)

    

   Microsoft ECC Root Certificate Authority 2017
   (Thumbprint: 999a64c37ff47d9fab95f14769891460eec4c3c5)

   Example:
   Downloading certificates using Linux wget utility:

$ mkdir certs
$ cd certs
$ wget https://cacerts.digicert.com/DigiCertGlobalRootG2.crt.pem
--2021-10-18 20:10:52--  https://cacerts.digicert.com/DigiCertGlobalRootG2.crt.pem
.
.
2021-10-18 20:10:53 (16.5 MB/s) - ‘DigiCertGlobalRootG2.crt.pem’ saved [1294/1294]

$ wget https://cacerts.digicert.com/DigiCertGlobalRootG3.crt.pem
--2021-10-18 20:32:21--  https://cacerts.digicert.com/DigiCertGlobalRootG3.crt.pem
.
.
2021-10-18 20:32:21 (41.1 MB/s) - ‘DigiCertGlobalRootG3.crt.pem’ saved [839/839]

$ wget  https://www.microsoft.com/pkiops/certs/Microsoft%20RSA%20Root%20Certificate%20Authority%202017.crt
--2021-10-18 20:31:44--  https://www.microsoft.com/pkiops/certs/Microsoft%20RSA%20Root%20Certificate%20Authority%202017.crt
.
.
2021-10-18 20:31:45 (73.2 MB/s) - ‘Microsoft RSA Root Certificate Authority 2017.crt’ saved [1452/1452]

$ wget  https://www.microsoft.com/pkiops/certs/Microsoft%20ECC%20Root%20Certificate%20Authority%202017.crt
--2021-10-18 20:31:16--  https://www.microsoft.com/pkiops/certs/Microsoft%20ECC%20Root%20Certificate%20Authority%202017.crt
.
.
2021-10-18 20:31:16 (15.7 MB/s) - ‘Microsoft ECC Root Certificate Authority 2017.crt’ saved [605/605]

admin@linux:~/certs$ ls -l
total 155
-rw-rw-rw-. 1 admin admin  178 Oct 18 20:32 cert.list
-rw-rw-rw-. 1 admin admin 1294 Dec  6  2017 DigiCertGlobalRootG2.crt.pem
-rw-rw-rw-. 1 admin admin  839 Sep 22 12:36 DigiCertGlobalRootG3.crt.pem
-rw-rw-rw-. 1 admin admin  605 Jan 22  2020 Microsoft ECC Root Certificate Authority 2017.crt
-rw-rw-rw-. 1 admin admin 1452 Jan 22  2020 Microsoft RSA Root Certificate Authority 2017.crt

3. Convert two Root Certificates to .pem format using below commands.

admin@linux:~/certs$ openssl x509 -inform der -in Microsoft\ ECC\ Root\ Certificate\ Authority\ 2017.crt -out ms-ecc-root.pem
admin@linux:~/certs$ openssl x509 -inform der -in Microsoft\ RSA\ Root\ Certificate\ Authority\ 2017.crt -out ms-rsa-root.pem
admin@linux:~/certs$ ls -l
total 155
-rw-rw-rw-. 1 admin admin  178 Oct 18 20:32 cert.list
-rw-rw-rw-. 1 admin admin 1294 Dec  6  2017 DigiCertGlobalRootG2.crt.pem
-rw-rw-rw-. 1 admin admin  839 Sep 22 12:36 DigiCertGlobalRootG3.crt.pem
-rw-rw-rw-. 1 admin admin  605 Jan 22  2020 Microsoft ECC Root Certificate Authority 2017.crt
-rw-rw-rw-. 1 admin admin 1452 Jan 22  2020 Microsoft RSA Root Certificate Authority 2017.crt
-rw-rw-rw-. 1 admin admin  875 Oct 18  2021 ms-ecc-root.pem
-rw-rw-rw-. 1 admin admin 2021 Oct 18  2021 ms-rsa-root.pem

4. Copy .pem certificate files on the Data Domain system.

admin@linux:~/certs$ scp *.pem sysadmin@dd01.example.com:/ddr/var/certificates/
DigiCertGlobalRootG2.crt.pem             100% 1294    13.6KB/s   00:00
DigiCertGlobalRootG3.crt.pem            100%  839     8.8KB/s   00:00
ms-ecc-root.pem                         100%  875     9.2KB/s   00:00
ms-rsa-root.pem                         100% 2021    21.2KB/s   00:00

5. Install the certificate as follows:

adminaccess certificate import ca application cloud file <file-name>
Certificates should be stored in /ddr/var/certificates before you run the adminaccess command.

admin@linux:~/certs$ ssh sysadmin@dd01.example.com

sysadmin@dd01# adminaccess certificate import ca application cloud file DigiCertGlobalRootG2.crt.pem

The SHA1 fingerprint for the imported CA certificate is:
DF:3C:24:F9:BF:D6:66:76:1B:26:80:73:FE:06:D1:CC:8D:4F:82:A4

        Do you want to import this certificate? (yes|no) [yes]: y
CA certificate imported for application(s) : "cloud".
sysadmin@dd01# adminaccess certificate import ca application cloud file DigiCertGlobalRootG3.crt.pem

The SHA1 fingerprint for the imported CA certificate is:
7E:04:DE:89:6A:3E:66:6D:00:E6:87:D3:3F:FA:D9:3B:E8:3D:34:9E

        Do you want to import this certificate? (yes|no) [yes]: y
CA certificate imported for application(s) : "cloud".
sysadmin@dd01# adminaccess certificate import ca application cloud file ms-ecc-root.pem

The SHA1 fingerprint for the imported CA certificate is:
99:9A:64:C3:7F:F4:7D:9F:AB:95:F1:47:69:89:14:60:EE:C4:C3:C5

        Do you want to import this certificate? (yes|no) [yes]: y
CA certificate imported for application(s) : "cloud".
sysadmin@dd01# adminaccess certificate import ca application cloud file ms-rsa-root.pem

The SHA1 fingerprint for the imported CA certificate is:
73:A5:E6:4A:3B:FF:83:16:FF:0E:DC:CC:61:8A:90:6E:4E:AE:4D:74

        Do you want to import this certificate? (yes|no) [yes]: y
CA certificate imported for application(s) : "cloud".

6. Check the new certificate information from the DD DDVE command line:

sysadmin@ddve# sysadmin@ddve# adminaccess cert show
Subject                                         Type            Application   Valid From                 Valid Until                Fingerprint
---------------------------------------------   -------------   -----------   ------------------------   ------------------------   -----------------------------------------------------------
dd01.example.com             host            https         Tue Mar 26 10:38:34 2019   Wed Jan 31 10:48:38 2024   30:78:FE:93:DF:2F:9D:B5:08:D7:EC:5E:9E:89:E2:BD:16:13:E1:BA
dd01.example.com             ca              trusted-ca    Wed Mar 27 17:38:34 2019   Wed Jan 31 10:16:38 2024   CB:9D:64:39:56:48:FB:58:C6:93:40:FB:29:91:56:9A:BD:08:7A:C8
Baltimore CyberTrust Root                       imported-ca     cloud         Fri May 12 11:46:00 2000   Mon May 12 16:59:00 2025   D4:DE:20:D0:5E:66:FC:53:FE:1A:50:88:2C:78:DB:28:52:CA:E4:74
DigiCert Global Root G2                         imported-ca     cloud         Thu Aug  1 05:00:00 2013   Fri Jan 15 04:00:00 2038   DF:3C:24:F9:BF:D6:66:76:1B:26:80:73:FE:06:D1:CC:8D:4F:82:A4
DigiCert Global Root G3                         imported-ca     cloud         Thu Aug  1 05:00:00 2013   Fri Jan 15 04:00:00 2038   7E:04:DE:89:6A:3E:66:6D:00:E6:87:D3:3F:FA:D9:3B:E8:3D:34:9E
Microsoft ECC Root Certificate Authority 2017   imported-ca     cloud         Wed Dec 18 15:06:45 2019   Fri Jul 18 16:16:04 2042   99:9A:64:C3:7F:F4:7D:9F:AB:95:F1:47:69:89:14:60:EE:C4:C3:C5
Microsoft RSA Root Certificate Authority 2017   imported-ca     cloud         Wed Dec 18 14:51:22 2019   Fri Jul 18 16:00:23 2042   73:A5:E6:4A:3B:FF:83:16:FF:0E:DC:CC:61:8A:90:6E:4E:AE:4D:74
---------------------------------------------   -------------   -----------   ------------------------   ------------------------   -----------------------------------------------------------

7. If any of the certificates were accidentally added for an "Application" other than "cloud," delete it from the Certification Authority certificate under Access Management UI.
   

   Note: Do not delete old "Baltimore Cyber-Trust Root" certificate.

For a Data Domain system that is configured with a Cloud Tier file system, a restart may be required to reestablish connection with Cloud Units. Arrange for downtime and run the following command to restart file system:
#filesys restart

For Data Domain system running on Azure Platform, reboot DDVE:
#system reboot