PowerProtect: While processing specific NAS file share, the protection job stops responding

NAS protection job fails while processing one of the Network File System (NFS) file shares. Attempt to cancel the job fails to stop protection job.

Protection jobs can only be stopped or marked with a failed state using following options:

• Restart protection engine that is running the protection job.
• Terminate pod within protection engine. 

If the status of the pod within protection engine is checked, it would show output similar to the following:

protectionengine:/home/admin # docker ps
CONTAINER ID   IMAGE                          COMMAND             CREATED         STATUS                     PORTS     NAMES
350d7c35417b   dpd/vproxy/nas-agent:bundled   "nasagentrestsvr"   5 minutes ago   Up 5 minutes (unhealthy)   443/tcp   f81ca6f3-2d2c-4049-a2ee-6647fed46dbe


protectionengine:/home/admin # docker inspect --format "{{json .State.Health }}" f81ca6f3-2d2c-4049-a2ee-6647fed46dbe 
{"Status":"unhealthy","FailingStreak":193,"Log":[{"Start":"2024-02-20T15:51:47.774259426-10:00","End":"2024-02-20T15:51:47.826684429-10:00","ExitCode":-1,"Output":"OCI runtime exec failed: exec failed: unable to start container process: unable to setup user: stat /dev/null: no such file or directory: unknown"},\{"Start":"2024-02-20T15:51:49.839557692-10:00","End":"2024-02-20T15:51:49.891627404-10:00","ExitCode":-1,"Output":"OCI runtime exec failed: exec failed: unable to start container process: unable to setup user: stat /dev/null: no such file or directory: unknown"},\{"Start":"2024-02-20T15:51:51.904688325-10:00","End":"2024-02-20T15:51:51.955100773-10:00","ExitCode":-1,"Output":"OCI runtime exec failed: exec failed: unable to start container process: unable to setup user: stat /dev/null: no such file or directory: unknown"},\{"Start":"2024-02-20T15:51:53.967149506-10:00","End":"2024-02-20T15:51:54.018310623-10:00","ExitCode":-1,"Output":"OCI runtime exec failed: exec failed: unable to start container process: unable to setup user: stat /dev/null: no such file or directory: unknown"},\{"Start":"2024-02-20T15:51:56.031730767-10:00","End":"2024-02-20T15:51:56.092704584-10:00","ExitCode":-1,"Output":"OCI runtime exec failed: exec failed: unable to start container process: unable to setup user: stat /dev/null: no such file or directory: unknown"}]} 

This issue occurs while processing files shares with the following names:

bin, data, etc, lib, mnt, proc, run, selinux, sys, tmp, var, boot, dev, home, lib64, opt, root, sbin, srv, usr

Backup process crates directory with the name of the share on the root file system of the container.

This directory is used as the mountpoint to access the share after the mount is successful. 

When the name of the share is from the list below, the NAS agent uses the default directory. This leads to conflict.

bin, data, etc, lib, mnt, proc, run, selinux, sys, tmp, var, boot, dev, home, lib64, opt, root, sbin, srv, usr

The impact of different folders can be different on the protection jobs. Sometimes the job may fail in other cases because pod would become unhealthy. The job continues to run. 

In the case of "/dev" file share, the name conflicts with the default set of top-level directory that is "dev". Hence, after mounting the share any of the existing references to the directory /dev has broken. Which results in an error:

container status and error "stat /dev/null: no such file or directory: unknown"

• Change the NAS share name to something else other than
   

  bin, data, etc, lib, mnt, proc, run, selinux, sys, tmp, var, boot, dev, home, lib64, opt, root, sbin, srv, usr

• Create a second copy of the file system with a different name and back that up instead of names that can cause conflict.

• Re-export the path as SMB/CIFS using a non-conflicting name (in the below above) and this requires support for multi-protocol enabled for on the array. This allows files to be backed up however, it would be using Common Internet File System (CIFS) and not the Network File system (NFS).