Data Domain Cloud tier : integrating data domain with amazon aws S3

The following article walks you through the needed steps to configure Data Domain cloud tier capabilities with amazon aws S3.
This guide is mainly divided into 4 main parts:

• Adding the required amazon aws user credentials from aws "IAM"  
• Importing the CA certificate to enable the communication between Data Domain and S3
• Adding the cloud unit from Data Domain 
• Naming of the cloud unit 

First :Adding "IAM" user credentials

The first step in integrating Data Domain cloud tier with amazon AWS S3 is to add the required AWS user credentials from aws "IAM" . This user credentials will be imported to the data domain system to authorize the communicate with the amazon S3 

The AWS user credentials must have permissions to :

• Create and delete buckets
• Add, modify, and delete files within the buckets they create.

S3FullAccess is preferred, but these are the minimum requirements :

• CreateBucket 
• ListBucket
• DeleteBucket
• ListAllMyBuckets
• GetObject
• PutObject
• DeleteObject

A. Go to https://aws.amazon.com/ and log in to the AWS console or create a new account if this is your first time:


B. From the top left corner choose services, and search for IAM (AWS Identity and Access Management ), so we can create and manage AWS users and groups, and use permissions to allow and deny their access to AWS resources:


C. From the IAM page select "users" from the left menu then select " add user":


D. Give your new user a name, for example: "DD_S3_cloudtier" .
Select the access type to give it programmatic access, then click Next:
 

E. Give this user the required permissions to use S3 resources. Select add user to group , and then select create group:


F. Give a unique name for the group. For example: "S3FullAccess_DD_cloudtier" and then search for "AmazonS3FullAccess". When the option appears in the result menu select it and then click Create group: 


G. You will be prompted back to previous menu. Select the group we just created "S3FullAccess_DD_cloudtier" then click Next Tags: 


H. On the Review menu, double check that the details you entered are correct then click "Create user ": 


I. we reach an important page:
You have now the user "access key ID" and "secret access key". You will use them to integrate the Data Domain with your S3 resources. Click "download .csv" and save this CSV file in a secure place and copy the access key ID and secret access key because we will use them in Data Domain:



Second: Importing CA certificate
You must import the CA certificate to enable the communication between your Data Domain system and amazon S3.

A. To download the AWS root certificate, go to https://www.digicert.com/digicert-root-certificates.htm and select the Baltimore CyberTrust Root certificate:

 

• If your downloaded certificate has a .CRT extension, it must be converted to a PEM-encoded certificate. If so, use OpenSSL to convert the file from .crt format to .pem. For example, openssl x509 -inform der -in BaltimoreCyberTrustRoot.crt -out BaltimoreCyberTrustRoot.pem.
• You can know more about how to convert the certificate to PEM from the following KB article: https://support.emc.com/kb/488482

B. Go to the Data Domain GUI and follow the following procedure: 

• 1. Select Data Management > File System > Cloud Units.
• 2. In the tool bar, click Manage Certificates. The Manage Certificates for Cloud dialog is displayed.
• 3. Click Add.
• 4. Select one of these options:
  • I want to upload the certificate as a .pem file.

•  I want to copy and paste the certificate text.

                   Copy the contents of the .pem  file to your copy buffer.
                   Paste the buffer into the dialog.

• 5. Click Add.

We are done with adding the CA certificate. Next we are going to add our S3 cloud unit from Data Domain GUI. 

Third:  Adding the clout unit to Data Domain
Here is a quick comparison of some of the differences between DDOS releases and their cloud tier options available:
 

DDOS Versio	Capabilites
6.0	Only supports "S3 standart storage" class Doesn't have a cloud provider verification method  Doesn't support large object size feature
6.1	Supports "standard" and "Standard-Infrequent Access (S3 Standard-IA)" storage classes 6.1.1.5 >= : Have the cloud provider verification method  supports large object size feature
6.2	Supports "Standard" , "Standard-IA" and "One Zone-Infrequent Access (S3 One Zone-IA)" have the cloud verification method  supports large object size feature

 
from Data domain GUI , follow this procedure to add the S3 cloud unit:

• 1. Select Data Management > File System > Cloud Units.
• 2. Click Add. The Add Cloud Unit dialog is displayed.
• 3. Enter a name for this cloud unit. Only alphanumeric characters are allowed. The remaining fields in the Add Cloud Unit dialog pertain to the cloud provider account.
• 4. For Cloud provider, select Amazon Web Services S3 from the drop-down list.
• 5. Select the storage class from the drop-down list. Based on the version of the DDOS you will find different options based on the table above.

           Learn more details about different supported S3 storage classes from the following link to choose the storage class best suitable for your backup needs:
           https://aws.amazon.com/s3/storage-classes/
           

• 6. Select the appropriate Storage region from the drop-down list.
• 7. Enter the provider Access key "as password text", the one we obtained from amazon IAM in step 1. 
• 8. Enter the provider Secret key "as password text",the one we obtained from amazon IAM in step 1.
• 9. Ensure that port 443 (HTTPS) is not blocked in firewalls. Communication with the AWS cloud provider occurs on port 443.
• 10. If an HTTP proxy server is required to get around a firewall for this provider, click Configure for HTTP Proxy Server. Enter the proxy hostname, port, user, and password.

 

• 11. if you are having DDOS >= 6.1.1.5 then click the cloud verification button.

         More details about Data Domain cloud verification tool could be found here: https://support.emc.com/kb/521796
          
         If your DDOS version is 6.0 then click add as the cloud verification option is not available in this release. 

• 12. Click Add. The File System main window now displays summary information for the new cloud unit as well a control for enabling  and disabling the cloud unit.
•  

Note: You can update the S3 cloud unit access key and secret access key ID afterwards from Data Domain GUI easily if needed.


Third:  Naming of the cloud unit
If we go back now to amazon S3, we will find that the Data Domain system created 3 buckets for this cloud unit:


The naming convention for the 3 buckets are as follows:

• A 16 character hexadecimal string.
• A dash character ('-').
• Another 16 character hexadecimal string, the hexadecimal string is unique for this cloud unit.
• Another dash character ('-').
• The buckets will end with the string '-d0', '-c0' and '-m0'.
• The bucket ending with the string '-d0' is used for data segments.
• The bucket ending with the string '-c0' is used for configuration data.
• The bucket ending with the string '-m0' is used for metadata.

For more details about the naming of the cloud units check the following KB article: https://support.emc.com/kb/487833

You are now done with creating S3 cloud unit that is integrated with your Data Domain system, and are ready to start applying data movement policies for your Mtrees to migrate the data to the newly created cloud tier unit .

• For better cloud tier capabilities, we would recommend upgrading to  DDOS 6.1.2.0 and later to benefit from "Large Object Size for Cloud Tier" feature added in theses releases for better cost and space optimization.

           Check the following KB for more details :https://support.emc.com/kb/522706
 

• How to remove cloud tier unit from Data Domain: Check the following KB for more details :https://support.emc.com/kb/488612

 

• Configuring the Data movement policy, and more details about cloud tier:

           Check the following admin guide (starting from page 427 for the data-movement policy configuration)  :
           https://support.emc.com/docu78746_Data-Domain-Operating-System-6.0-Administration-Guide.pdf?language=en_US
 

• Cloud tier cleaning: Check the following KB article for more details :https://support.emc.com/kb/487657