NVP vProxy: How To Perform A File Level Recovery When FLR Is Failing or Not Supported By the VM

The following procedure can be used as a workaround or solution when File Level Recovery (FLR) is failing and a file restore is urgently needed or when limitations impact an FLR. For a full list of VM FLR limitations, see the NetWorker VMware Integration Guide for your NetWorker version. These guides are available through Dell Support at https://www.dell.com/support/home/product-support/product/networker/docs

This method requires performing an Instant Recovery. This creates an NFS export on the Data Domain and access is granted to the ESXi host specified in the recover options. This means that the VM's configuration or disk files are available to the ESXi host specified. You can then mount disks from the Instant Access Restore to the original VM and copy the needed files. The following are requirements for performing an Instant Access Restore:

• Ensure that you provide the management credentials for the Data Domain resource before you initiate the recovery. This field can be found under Devices-->Data Domain Systems-->NSR Data Domain Properties (Edit properties of DD).

• Ensure you do not perform an instant recovery of virtual machines in resource pools and other similar containers that are part of a running protection group.
• Do not perform an Instant Recovery of a VM simultaneously that it is being backed up.
• Ensure that the free space on the Data Domain system is equal to or greater than the total disk size of the virtual machine being restored, as the restore does not factor in the space that is required after deduplication occurs. If there is insufficient disk space, an error appears indicating "Insufficient disk space on datastore," and creation of the target virtual machine fails.
• If VM is Linux-based there are some additional considerations depending on what the disk is (operating system vs. data disk, specific Linux operating system). The Linux System administrator must be engaged to ensure that devices are mounted and mapped properly. If the data resides on a LVM spanning multiple disks, this workaround cannot be used. 

   

Procedure:

1. Log in to the vSphere Web Client and select the VM you plan on mounting the Instant Recovery Disks to. The Host field in the VM's Summary tab shows the current ESXi hostname or IP address.

5. Proceed through the rest of the Instant Recovery. Once complete, you see it running in the NMC. This stays here until you manually delete the restore session. You can close the Recover Configuration window by clicking Finish, this does not stop the recovery.

6. In the vSphere Web Client, you see that a new VM is created by the Instant Access Restore; this VM can be ignored. Right-click the VM you want to mount the disks to and select edit settings. 
7. Select Add-->Existing Disk-->Add.
8. Navigate through the EMC-Recover-vproxy_name datastore and select the VMDK you want to mount to the original VM.

9. Log in to the VM you mounted the disk to and open Disk Management, the disk may be offline, set it to Online.

10. You can use Windows File Explorer to copy or paste files or directories from the mounted restore drive to the original location.

11. When you are done with the disk, you can go back into the VM settings and detach or delete the disk. Do not delete files from the datastore. You can confirm which disk it is by the Disk File path:

12. In NMC, right-click the running Instant Access Restore session, select stop, click yes on the delete prompt to delete Instant Access Restore VM and NFS datastore.

The below provides a basic example of mounting a single Linux VMDK after performing steps 1 through 8 above. In the following example, we have a Red Hat Enterprise Linux system with /data on Disk 2 (seen as /dev/sdb1 on system): df -h

[root@vmrhel7 ~]# df -h
Filesystem                     Size  Used Avail Use% Mounted on
devtmpfs                       1.9G     0  1.9G   0% /dev
tmpfs                          1.9G     0  1.9G   0% /dev/shm
tmpfs                          1.9G  8.9M  1.9G   1% /run
tmpfs                          1.9G     0  1.9G   0% /sys/fs/cgroup
/dev/mapper/rhel_vmrhel7-root   14G  3.4G   11G  26% /
/dev/sda1                     1014M  233M  782M  23% /boot
tmpfs                          379M     0  379M   0% /run/user/0
/dev/sdb1                       40G   34M   40G   1% /data

After completing steps 1-8 from the resolution field, we now see an additional disk /dev/sdc attached to the system: lsblk -o NAME,FSTYPE,LABEL,SIZE,MOUNTPOINT

[root@vmrhel7 ~]# lsblk -o NAME,FSTYPE,LABEL,SIZE,MOUNTPOINT
NAME                  FSTYPE      LABEL  SIZE MOUNTPOINT
sda                                       16G 
├─sda1                xfs                  1G /boot
└─sda2                LVM2_member         15G 
  ├─rhel_vmrhel7-root xfs               13.4G /
  └─rhel_vmrhel7-swap swap               1.6G [SWAP]
sdb                                       40G 
└─sdb1                xfs                 40G /data
sdc                                       40G 
└─sdc1                xfs                 40G 
sr0                                     1024M 

Create a temporary folder to mount the disk to: mkdir /tmp/flr
Mount the disk to the folder: mount -o rw,nouuid /dev/sdc1 /tmp/flr

[root@vmrhel7 ~]# mount /dev/sdc1 /tmp/flr                  
mount: wrong fs type, bad option, bad superblock on /dev/sdc1,
       missing codepage or helper program, or other error

       In some cases useful info is found in syslog - try
       dmesg | tail or so.

[root@vmrhel7 ~]# dmesg | tail
[12582.452435] sd 0:0:2:0: [sdc] Mode Sense: 61 00 00 00
[12582.452467] sd 0:0:2:0: [sdc] Cache data unavailable
[12582.452468] sd 0:0:2:0: [sdc] Assuming drive cache: write through
[12582.454682]  sdc: sdc1
[12582.455066] sd 0:0:2:0: [sdc] Attached SCSI disk
[13036.466924] XFS (sdc1): Filesystem has duplicate UUID ccd31faa-3ceb-47ff-8f64-ad999bb1ab5a - can't mount
[13053.416540] XFS (sdc1): Filesystem has duplicate UUID ccd31faa-3ceb-47ff-8f64-ad999bb1ab5a - can't mount
[13208.507880] XFS (sdc1): Filesystem has duplicate UUID ccd31faa-3ceb-47ff-8f64-ad999bb1ab5a - can't mount
[13464.609162]  sdc: sdc1
[13494.625617] XFS (sdc1): Filesystem has duplicate UUID ccd31faa-3ceb-47ff-8f64-ad999bb1ab5a - can't mount

[root@vmrhel7 ~]# mount -o rw,nouuid /dev/sdc1 /tmp/flr        
[root@vmrhel7 ~]# 

Note: If the disk is being mounted to the same system, it was backed up on the mount may fail because the UUID of the disk is the same as another disk on the system. As we are only recovering files and not adding this disk to the permanent file system, you can mount with nouuid.

You can now copy the needed data from the FLR mountpoint to a location of your choosing: cp /tmp/flr/path/to/file-or-dir /path/to/destination/dir

[root@vmrhel7 ~]# cp /tmp/flr/sysctl.conf ~/flr/ 

[root@vmrhel7 ~]# ls -l flr/ 
total 4 -rw-r--r--. 1 root root 449 Jun 16 15:36 sysctl.conf

See steps 11 and 12 to remove the disk and delete the Instant restore VM instance.