Symptoms
Data Domain Virtual Edition(DDVE) is a software-only storage Appliance, which can be either on-premises or cloud deployed. Active Tier on Object Storage (ATOS) is another name for cloud deployed DDVE.
On premises, DDVE supports VMware, Hyper-V, KVM, and VxRail.
In the cloud, Data Domain Virtual Edition runs in:
- The Amazon Web Services (AWS) (cloud and gov cloud)
- Azure (cloud and gov cloud)
- VMware Cloud on AWS cloud platforms
- Google Cloud Platform (GCP)
DDVE on Cloud is not the same as a Cloud Tier. Cloud Tier is a different product that can only be deployed on on-premises Hardware DDs and DDVEs.
If a DDVE deployed on the Cloud fills up its Local-Metadata Storage, the following alert is issued:
EVT-SPACE-00004: Space usage in metadata storage has exceeded 100% threshold.
Cause
DDVE deployed on Cloud (ATOS) supports two types of data storage:
- Block Storage (Used for Data and Metadata, or Metadata on Object Storage enabled DDVEs aka ATOS).
- Hot Blob Storage is also known as Object Storage (Used only for Data)
Metadata Disks on an ATOS deployed DDVE are populated with: Index, LP segment references, CMETA (Metadata) containers, and Directory manager (DM) data
Most deployments are using (Hot Blob or Object Storage) as that is the current recommendation on Cloud Deployments. All the data is written using Object Storage, but 10% of space is defined as Block Storage for the DD Filesystem metadata. The size of the local-metadata is a speculation based on the size of the Licensed Capacity. At deployment time, 10% Metadata usage is estimated for a 10x deduplication factor. A DDVE with a 100% Metadata Storage usage cannot ingest backups.
Due to the following scenarios, the DDVE may run out of local-metadata storage space:
- The deduplication is higher than 10x, as such the metadata requirements are higher than what was deployed in metadata size(default 10%).
- A large number or old snapshots can increase the metadata size
A larger amount of metadata may be noticed in ATOS DDVEs acting as Destinations for File Replication (Controlled Clone Replication CCR /Managed File Replication MFR):
- If not all base files are replicated, the destination ATOS DDVE has a higher usage in metadata.
- In this scenario Replication is NOT taking full advantage of the Virtual Synthetics optimizations, as such needing more space to store metadata.
- This type of problem was noticed in NW CCR, where it does not seem all the base files were being replicated. This causes the destination to have a higher metadata usage.
Resolution
The solution for this issue would be to add further disk/s as Local Metadata Storage.
Only AWS and Azure allow metadata disk size expansion when the the maximum allowed number of disks have already been allocated. The metadata capacity for existing metadata disks can increased 1 TiB increments. This does not apply to the first metadata disk. See the "Expand metadata storage" section in the guide document for the details.
IMPORTANT INFORMATINO WHEN ADDING DISKS FOR METADATA:
- Using different-sized metadata disks creates an index imbalance.
- Each metadata disk added to the active tier should have its own Spindle Group, the software automatically does this. The maximum known number of spindle groups is 16.
- If going over the max number of spindle groups (16), metadata disks will the same spindle group. I/O against disks on the same spindle groups is sequential.
- See the manual for the recommended sizes, most recommend a 1 TiB increment, higher sizes are possible depending on Cloud provider.
- Only expand metadata disks for AWS and Azure. Expanding metadata disks in any provider makes the DDVE unusable with corruption, or having the expanded space unused.
-
Review the Metadata Usage, if it reached 100%, it needs more disks assigned.
- The example below is provided, some outputs may differ depending on Cloud Provider, see the Install and Admin Manual:
- Display the metadata Usage.
# filesys show space tier active local-metadata
--------------------------
Active Tier: local-metadata usage
Size GiB Used GiB Avail GiB Use%
-------- -------- --------- ------
1293.0 1291.5 1.5 100.0% -> We can see Metadata space is full.
-------- -------- --------- ------
-
The process of adding more Metadata disks differs depending on the Cloud Provider, see each of the Cloud Provider manuals at the bottom.
- Display which disks are used.
# disk show hardware
------------------
Disk Slot Manufacturer/Model Firmware Serial No. Capacity Type
(pci/idx)
---- --------- -------------------- -------- ---------- --------- -----
dev1 -/a Virtual BLOCK Device n/a (unknown) 250.0 GiB BLOCK -> DDOS Disk
dev2 -/b Virtual BLOCK Device n/a (unknown) 10.0 GiB BLOCK -> NVRAM disk
dev3 -/c Virtual BLOCK Device n/a (unknown) 1.0 TiB BLOCK -> Currently used for Metadata
dev4 -/d Virtual BLOCK Device n/a (unknown) 1.0 TiB BLOCK -> Currently used for Metadata
---- --------- -------------------- -------- ---------- --------- -----
4 drives present.
- Go to the AWS/GCP/AZURE console and add storage to the DDVE. In this case, a 1TiB disk is added. It is seen as dev5, and remember not to expand or touch any of the other existing disks.
MANUALS for DDVE on Cloud (Only posting below for DDVE V4)
DDVEV4 on Microsoft Azure Install and Admin Manual.
https://support.emc.com/docu91981_Data-Domain-Virtual-Edition-4.0-with-DD-OS-6.2.0.10-in-Azure-Installation-and-Administration-Guide.pdf?language=en_US
DDVEV4 on Google Cloud Platform(GC)
https://support.emc.com/docu91982_Data-Domain-Virtual-Edition-4.0-with-DD-OS-6.2.0.10-in-Google-Cloud-Platform-(GCP)-Installation-and-Administration-Guide.pdf?language=en_US
DDVE4 on Amazon web Services(AWS), Install, and Admin Manual
https://support.emc.com/docu91980_Data_Domain_Virtual_Edition_4.0_with_DD_OS_6.2.0.10_in_Amazon_Web_Services_(AWS)_Installation_and_Administration_Guide.pdf?language=en_US
ATTENTION: New and Updated Install or Admin Manuals may exist at the time of reading this article.
- At this stage assuming a new 1 TiB disk was added, this is seen on the output as disk dev5.
# disk show hardware
------------------
Disk Slot Manufacturer/Model Firmware Serial No. Capacity Type
(pci/idx)
---- --------- -------------------- -------- ---------- --------- -----
dev1 -/a Virtual BLOCK Device n/a (unknown) 250.0 GiB BLOCK -> DDOS Disk
dev2 -/b Virtual BLOCK Device n/a (unknown) 10.0 GiB BLOCK -> NVRAM disk
dev3 -/c Virtual BLOCK Device n/a (unknown) 1.0 TiB BLOCK -> Currently used for Metadata
dev4 -/d Virtual BLOCK Device n/a (unknown) 1.0 TiB BLOCK -> Currently used for Metadata
dev5 -/e Virtual BLOCK Device n/a (unknown) 1.0 TiB BLOCK -> Newly added disk for metadata, unused.
---- --------- -------------------- -------- ---------- --------- -----
- Add the new disk to the active Tier. dev5 for this example:
# storage add tier active dev5
- Expand the file system.
# filesys expand
-
Confirm the FS which was read-only due to lack of metadata space can now be written to.
- Check that backups are now working fine
- Check that incoming replication resumed and traffic may be seen.
This content is translated in other languages:
- https://downloads.dell.com/TranslatedPDF/PT-BR_KB537488.pdf
- https://downloads.dell.com/TranslatedPDF/ZH-CN_KB537488.pdf
- https://downloads.dell.com/TranslatedPDF/ES_KB537488.pdf
- https://downloads.dell.com/TranslatedPDF/DE_KB537488.pdf
- https://downloads.dell.com/TranslatedPDF/FR_KB537488.pdf
- https://downloads.dell.com/TranslatedPDF/IT_KB537488.pdf
- https://downloads.dell.com/TranslatedPDF/JA_KB537488.pdf
- https://downloads.dell.com/TranslatedPDF/NL_KB537488.pdf
- https://downloads.dell.com/TranslatedPDF/KO_KB537488.pdf
- https://downloads.dell.com/TranslatedPDF/RU_KB537488.pdf
- https://downloads.dell.com/TranslatedPDF/PT_KB537488.pdf
- https://downloads.dell.com/TranslatedPDF/SV_KB537488.pdf
- The Filesystem expand can be done online.
- Metadata usage is directly proportional to the deduplication factor. Workloads with high deduplication factors create larger indexes, so the metadata needs are higher.
- Old and large snapshots hold data, and also increase the deduplication factor, and cause the system to run out of metadata sooner.
- There are only Metadata Requirement Guidelines for deployment, these are 10% of Capacity in Metadata disks, and assume a 10x deduplication factors. Higher deduplication ratios require more metadata disks.
- DD Support does not have local-metadata storage requirement guidelines for higher deduplication factors than 10x. Metadata Disks are recommended to be added in 1 TiB increments until a balance is reached between Filesystem Usage and Metadata Usage.
- There is no known method of predicting the future Metadata Usage based on potential deduplication factors.
- Index imbalance can also occur on disks of the same size. The most common reason would be that new disks have been incremented post metadata being nearly full. The older disks are still holding most of the metadata structures, with more I/O being requested. As older data is expired, the metadata should balance itself between the disks.
Affected Products
Data Domain, Data Domain Virtual Edition