Data Domain : Backup application integrated with Retention Lock may experience backup failures due to configuration issue

Some backup jobs fail on the backup client with messages such as the following :
 

8212 6df7 12/05 15:47:15 871396 [MEDIAFS    ] 3637866-3138214 Cannot set the access time of [/data/col1/Commvault/SUBDIR/CV_MAGNETIC/V_305788/CHUNK_18517307], error=0xECCC000D:{CQiFile::SetTimes(825)/ErrNo.13.(Permission denied)}
8212 6df7 12/05 15:47:15 871396 [MEDIAFS    ] 3637866-3138214 Cannot mark the file [/data/col1/Commvault/SUBDIR/CV_MAGNETIC/V_305788/CHUNK_18517307] as read only.

The "read-only" mode for backups and images is how this particular backup software calls the DD backend feature that allows an administrator to set a period of time during which the file in the backend may not be modified or deleted, for protection against accidental or malicious data deletion. This feature is what is called the Data Domain Retention Lock (RL for short).

On the DD side, logs show the following for the same storage unit, subdirectory and backup file :

12/05 07:47:47.820284 [7f1bc842a000] Attempt to set atime of 16adcf:0:16addb:0:7d70db86:6256fe81:0 to larger than maximum retention period of mtree.
12/05 07:47:47.820289 [7f1bc842a000] ERROR: FM fm_dm1_setattr:1408 - fm_dm1_setattr_intern failed
12/05 07:47:47.820533 [7f1bcdf19d90] ddboost-<backupsoftware.example.com-56892>: ddboost_api ERROR: ddp_utime() failed, su_name=Commvault, path_name=/SUBDIR/CV_MAGNETIC/V_305788/CHUNK_18517307, Err: 5034-nfs setattr failed (nfs: Permission denied)

The DD RL configuration for each individual MTree which has the feature enabled includes setting the minimum (Retention-lock min-retention-period) and maximum (Retention-lock max-retention-period) duration of locks that are allowed to be set on any of the files in the MTree. With DD RL the backup application has to individually set the lock on files, unless the DD Automatic Retention Lock (ARL) feature is enabled. The options for the MTree in the example were as follows :
 

Mtree: /data/col1/Commvault

Option                                      Value
-----------------------------------------   -----------
Retention-lock                              enabled
Retention-lock mode                         governance
Retention-lock uuid                         UUID1:UUID2
Retention-lock min-retention-period         720minutes
Retention-lock max-retention-period         35days
Retention-lock automatic-retention-period   not set
Retention-lock automatic-lock-delay         120minutes
Retention-lock indefinite-retention-hold    disabled
-----------------------------------------   -----------

This means for any file in the MTree, the RL may only be set 720 minutes from the current time (or longer), and 35 days from the current time (or shorter).In other words, with the configuration above a file may only be protected from modification or removal for a period of more than 12 hours, but less than 35 days. Any attempt by the backup application to set a lock (which is done by updating the file's atime, when using BOOST; through the "ddp_utime" call) for a shorter or a longer duration will result in the error presented above :

12/05 07:47:47.820284 [7f1bc842a000] Attempt to set atime of 16adcf:0:16addb:0:7d70db86:6256fe81:0 to larger than maximum retention period of mtree.

When the backup application knows how to use the DD RL feature it will wait for the backup to finish writing to the image in the backend and then eventually set the lock on the backup image (or images, as some software may use more than one file to store a single backup job). The the BOOST libraries will be used to call the "ddp_utime" to set the lock for a duration equal to the intended backup retention at the backup application level. This has two implications :

• If time is not in sync between the backup application and the DD, the backup application may calculate "X days from now" and obtain a date and time which is not exactly the same as that for the DD, which would result in the backup image locked for a shorter or longer period of time depending on the sign of the time difference
• If the intended backup retention is not aligned with the RL limits on the DD MTree, the backup application may attempt to set a lock too far in the future (for a period longer than "Retention-lock max-retention-period") and hence setting of the lock will be denied. If, for example, the backup application retention is 60 days with a "Retention-lock max-retention-period" set to 30 days in the DD, setting the lock will obviously fail

In situations in which the backup software retention is equal to the "Retention-lock max-retention-period" on the DD, any minor time differences may result in the lock setting being denied due to the time difference between the two hosts.

It is important that all hosts in the backup infrastructure have the correct time, and hence that they sync through NTP or (if applicable) Windows AD.

To avoid corner cases such as the one described it is a good practice that the "Retention-lock max-retention-period" in the RL-enabled MTree is set to slightly longer than the longest-retaining backup policy stored in that MTree. For example, if data retention is set to 35 days in backup application, setting the "Retention-lock max-retention-period" on the DD MTree used to store those policies to 36 or even 40 days is the right thing to do to avoid accidental failures to set RL.

Note having a "Retention-lock max-retention-period" higher than the backup images retention period is not an issue. If we had 100 days "Retention-lock max-retention-period" for a 35 days retention backup policy, after 35 days the images will be deleted by the application, and clean will dispose of their used space next time it runs. Only drawback is in the case of accidentally setting images with a longer lock, with RL Compliance, you will not be able to delete the files for longer than expected. So the recommendation is to set "Retention-lock max-retention-period" a bit longer, but not too much.