Error Code Event ID 4009 - Snapshot creation failed with error 0x80004005

Question

Hi, I have recently moved my clustered VMs to some new nodes (from Dell R710s to R740s). The VMs run on a 3 node Hyper-V 2012 R2 cluster. The VM files are stored on Dell EqualLogic SANs. Some VMs are on CSVs while others are on dedicated LUNs/volumes. The ones with dedicated LUNs are the ones that aren't working since the migration. Since moving these VMs to the new nodes, DPM has failed when trying to backup 2 out of 5 VMs. The VMs that fail to backup are the only 2 that are on their own dedicated LUNs (the other 3 VMs use CSV storage). At the moment the protection groups for each VM are reporting as 'Replica is inconsistent'. When I try to perform a consistency check, the job starts and the VM shows as 'Backing up' in Hyper-V Manager. However, this lasts for about 5 minutes before stopping and merging changes (and the AVHD file it created gets deleted). This repeats over and over again without anything actually happening. When I monitor the job in DPM I can see 0MB being transferred.  That said, each time the 'Backing up...' stops on Hyper-V, I see the following entry in the Application Logs in Event Viewer: It says there is insufficient snapshot reserve space but all of our EqualLogic volumes have no snapshot reserve, this is intentional. Nothing has changed on the storage since before we moved these VMs to new nodes. We simply connected the iSCSI connections to the new nodes. The new nodes are running HIT 5.0. Could this be something to do with the Auto-Snapshot Manager? I see this warning in Auto-Snapshot Manager. I'm not sure why DPM is suddenly unable to backup the VM files on these volumes now that they are connected to new machines.  Any ideas / help would be greatly appreciated.

Opgailey · Answer

@Anonymouswrote:

Hello,

I suspect that the EQL VSS H/W provider is enabled on those new nodes. DPM is going to call VSS to create a "snapshot" of the volume. VSS will search for a H/W provider for that disk. In effect asking 'who owns this disk?'. In a system w/o a 3rd party H/W provider it will use unallocated space on the filesystem to be backed up, using a hidden snapshot file. When you install HIT/ME it will install the EQL H/W provider automatically. So now DPM will call VSS, VSS will query the disk and see that it's an EQL device. VSS finds the H/W provider and sends a request to the array via iSCSI to the vss-control volume on the array to create an EQL snapshot on the array "aka hardware snapshot" instead of hidden file. Since you have no snapshot reserve the creation fails.

This is easy fix. Login as admin level user. At CLI. eqlvss /unregister

C:\Program Files\EqualLogic\bin>eqlvss /unregserver

You can confirm it has been removed with the following command:

C:\Program Files\EqualLogic\bin>vssadmin list providers

Try this on all the hosts and let us know if that resolves it.

Regards,

Don

You were spot on (once again). Your fix has worked like a charm.

Thanks for the detailed explanation and resolution. This is the second time you've helped me with EqualLogic queries in the last month.

Very much appreciated.

EqualLogic

Was this post helpful?