Skip to main content
Sign Out

Welcome to Dell

My Account
  • Place orders quickly and easily
  • View orders and track your shipping status
  • Enjoy members-only rewards and discounts
  • Create and access a list of your products
Sign In Create an Account Dell Financial Services Premier Sign In Partner Program Sign In
Contact Us

Your Dell.com Carts

NetWorker VMware Protection-vProxy: How to Enable Debug Logging

Summary: Changing the log level to 'debug' may be necessary in order to troubleshoot an issue with NetWorker VMware Protection (NVP) with vProxy Appliance solution (NVP-vProxy). This article explains how to change the log level. ...

This article applies to   This article does not apply to 
Check out resources for

Instructions

Watch this Video: How to Enable vProxy Debug Logging.

 


Usually, it is sufficient to analyze the policy logs of the NetWorker server in /nsr/logs/policy/[policy name], but occasionally it may be necessary to analyze the logs from the vProxy appliance itself and furthermore, to increase the logging level to debug in order to get a clearer idea of the reason for a failure. This is typically only a requirement once NetWorker support has been engaged and the support engineer has requested to debug set to troubleshoot as specific process. 

NVP-vProxy log files on the vProxy appliance are in the following directory /opt/emc/vproxy/runtime/logs.

There is a subdirectory for each of the principal processes which are used for NetWorker operations on the vProxy appliance: 
vbackupd

vflrd

vrapid

vrecoverd
For example, in the vbackupd subdirectory, you have the logs: 
[session-uuid].log             Processing details for a session
vbackupd-engine.log            Requests and problem events including error and panic messages from the session log.
vbackupd-boost.log             DD Boost logging.
vbackupd-snapmgr.log           Snapshot logging
3. vbackupd-vddk.log           VDDK logging -  VMware Virtual Disk Development Kit
Additional information about vProxy logging and collection is found in the article NVP-vProxy Triage Guide: NetWorker VMware Protection vProxy Triage and Log Overview


Applying Service Debug Method 1:

This method applies to vProxy's version 4.3.x and later. It can be used to enable debug on vProxy services (vrapid, vrecoverd, vbackupd, vflrd). More debugging can be enabled for DD Boost traffic, VDDK, and snapshot messaging (where applicable).
  1. Connect to the vProxy over SSH using the vProxy admin account.
  2. Switch to root:
sudo su -
  1. Change to the vProxy agent bin directory:
cd /opt/emc/vproxy/bin
  1. For the service you want to debug, run the following commands:

Backup Engine: vbackupd
Daemon Logging Entity Command Usage Example Log Locations
vbackupd Program (main engine) 
./vProxy_debug.sh -s -d vbackupd -p program -l debug
 
/opt/emc/vrproxy/runtime/logs/vbackupd/vbackupd-engine.log

Individual backup session logs: 

/opt/emc/vrproxy/runtime/logs/vbackupd/BackupVmSession-<session-id>.log
/opt/emc/vrproxy/runtime/logs/recycle/vbackupd/<DATE>/BackupVmSession-<session-id>.log
  DD Boost 
./vProxy_debug.sh -s -d vbackupd -p ddboost -l all
 
/opt/emc/vrproxy/runtime/logs/vbackupd/vbackupd-boost.log
  VDDK 
./vProxy_debug.sh -s -d vbackupd -p vddk -l 4
 
/opt/emc/vrproxy/runtime/logs/vbackupd/vbackupd-vddk.log
The VDDK also maintains private logs in the directory /tmp/vmware-root
       Snapshot Manager 
./vProxy_debug.sh -s -d vbackupd -p snapshotmgr -l debug
 
/opt/emc/vrproxy/runtime/logs/vbackupd/vbackupd-snapmgr.log

Recover Engine: vrecoverd
Daemon Logging Entity Command Usage Example Log Locations
vrecoverd Program (main entity) 
./vProxy_debug.sh -s -d vrecoverd -p program -l debug
 
/opt/emc/vrproxy/runtime/logs/vrecoverd/vrecoverd-engine.log

Individual Recover Session Logs: 

/opt/emc/vrproxy/runtime/logs/vrecoverd/RecoverVmSession-<session-id>.log
/opt/emc/vrproxy/runtime/logs/recycle/vrecoverd/<DATE>/RecoverVmSession-<session-id>.log
  DD Boost 
./vProxy_debug.sh -s -d vrecoverd -p ddboost -l all
 
/opt/emc/vrproxy/runtime/logs/vrecoverd/vrecoverd-boost.log
  VDDK 
./vProxy_debug.sh -s -d vrecoverd -p vddk -l 4
 
/opt/emc/vrproxy/runtime/logs/vrecoverd/vrecoverd-vddk.log
The VDDK also maintains private logs in directory /tmp/vmware-root

FLR Engine: vflrd
Daemon Logging Entity Command Usage Example Log Location
vflrd Program (Main Engine) 
./vProxy_debug.sh -s -d vflrd -p program -l debug
 
/opt/emc/vrproxy/runtime/logs/vflrd/vflrd-engine.log
Individual FLR Mount Session Logs: 
/opt/emc/vrproxy/runtime/logs/vflr/FlrMountSession-<session-id>.log
Individual FLR Browse Session Logs: /opt/emc/vrproxy/runtime/logs/vflr/FlrBrowseSession-<session-id>.log
Individual FLR Recover Session Logs: /opt/emc/vrproxy/runtime/logs/vflr/FlrRecoverSession-<session-id>.log
Completed logs are moved to 
/opt/emc/vrproxy/runtime/logs/recycle/vflrd
  DD Boost 
./vProxy_debug.sh -s -d vflrd -p ddboost -l all
 
/opt/emc/vrproxy/runtime/logs/vflrd/vflrd-boost.log
  VDDK 
./vProxy_debug.sh -s -d vflrd -p vddk -l 4
 
/opt/emc/vrproxy/runtime/logs/vrecoverd/vrecoverd-vddk.log
The VDDK also maintains private logs in directory /tmp/vmware-root

REST Server: vrapid
Daemon Logging Entity Command Usage Example Log Location
vrapid Program (Main Engine) 
./vProxy_debug.sh -s -d vrapid -p program -l debug
 
/opt/emc/vrproxy/runtime/logs/vrapid/vrapid-engine.log

The above commands show enabling debug, these commands can be rerun changing the -l option to reset to default:

Options:
  • For Program Log:
warn | info | trace (default) | debug
  • For DD Boost Log:
none | error | warn | info | debug | trace (default) | all
  • For VDDK Log:
0 = No logging
1 = Errors only
2 = Warnings and Errors
3 = Important information messages, errors, and warnings (default)
4 = Debug data plus everything else
  • For Snapshot Manager Log:
warn | info | trace (default) | debug


Applying Service Debug Method 2:

The available logging levels are none, error, warn, info, trace, and debug.

In order to set the vbackupd process into logging level debug:
  1. Log in to the vProxy Appliance as admin over SSH or console and switch to root:
sudo su -
  1. Go to the systemd directory:
cd /usr/lib/systemd/system/ 

Note: Each of the service files is found here: vbackupd.service, vfrld.service, vrapid.service, vrecoverd.service, and so on
  1. Open the service file that you want to apply debug on, for example:
vi vbackupd.service 
  1. Locate the line with ExecStart= and append with --program-log-level=debug
Other parameters: 
vProxy 2.x engine log level: { debug, trace, info, warn }.
--engine-log-level=string        

vProxy 3.x and later engine level: { debug, trace, info, warn } (default "trace")
--program-log-level=string (vbackupd)

Display version
--version
 
Note: The NVP-vProxy version is shown in the Configuration tab of the vProxy properties in the NetWorker Management Console > Devices > VMware Proxies > right click the vProxy appliance > Properties > Configuration > VM Proxy version.
 
  1. Save the changes to the file
  2. Reload the unit configuration file by typing the command:
systemctl daemon-reload
  1. Confirm in the NetWorker Management Console that no backups are running and then restart the vbackupd service:
systemctl restart vbackupd.service
The above process is applicable for the services vrapid, vrecoverd, vbackupd, vflrd.


Applying VDDK Debug:

  1. Log in to the vProxy appliance using SSH or console as admin and switch to root:
sudu su -
  1. Edit the VixDiskLib.config file to contain the following line:
vi /opt/emc/vproxy/conf/VixDiskLib.config vixDiskLib.transport.LogLevel=4

0 = No logging
1 = Errors only
2 = Warnings, and Errors
3 = Important information messages, errors, and warnings
4 = Debug data plus everything else
  1. Restart the relevant service:
systemctl restart [service-name].service 
For example:  
systemctl restart vrecoverd.service
Once you have set the debug logging level, you can confirm that there are messages in the relevant log file which start with the word Debug.

After completing the debugging process, you can reset the log level using the same procedure, using the word trace after the equal sign instead of debug. In order for this change to be immediately applicable, a daemon reload and restart is required as outlined above.


DD Boost Precert Debugging:

This process is done with other debugging methods outlined above.
  1. Log in to the vProxy appliance using SSH or console as admin and switch to root:
sudu su -
  1. Create a file ddboost_precert.ini under:
/: touch /ddboost_precert.ini
  1. Create the DDLog directory:
mkdir /DDLog
  1. Enable DD Boost debug for the service that you want to debug:
vbackupd: /opt/emc/vproxy/bin/vProxy_debug.sh -s -d vbackupd -p ddboost -l debug

vflrd: /opt/emc/vproxy/bin/vProxy_debug.sh -s -d vflrd -p ddboost -l debug

vrecoverd: /opt/emc/vproxy/bin/vProxy_debug.sh -s -d vrecoverd -p ddboost -l debug
  1. Confirm that the ddboost_precert.log is generated:
ls -l /DDLog
​​​​​​ screenshot of log confirmation
  1. When debug operations are no longer required, rerun the following command:
vProxy_debug.sh -l trace


vProxy Selection Logging:

This debugging can be used whenever a VM backup workflow is stuck with the VMs sitting in Waiting to Run. In this scenario, the backup job has started but no vProxy has been picked up to start the backup session. There is no logging of the vProxy selection process in standard logging. We can enable vmbackup_logging on the NetWorker server, which logs the vProxy selection process during the start of VMware Protection Policy. This can be useful when there is no communication issue between the NetWorker server and vProxy (Port 9090), and there are no errors regarding the vProxy being unavailable in the NMC or daemon.log.
 
  1. Open an Administrative PowerShell prompt (Windows) or a root SSH session (Linux) on the NetWorker server.
  2. Create a file without an extension called vmbackup_logging under the NetWorker /nsr/tmp directory:
Linux: 
touch /nsr/tmp/vmbackup_logging

Windows: 
New-Item -Path "C:\Program Files\EMC NetWorker\nsr\tmp\vmbackup_logging" -ItemType File
The above command assumes the default NetWorker install location was used. If NetWorker was installed under a different location, adjust the location of /nsr/tmp accordingly.
  1. Start the VM backup job.
  2. Monitor the rendered daemon.log for Vmproxy Queue and select vproxy messages.
Linux: /nsr/logs/daemon.log
Windows: C:\Program Files\EMC NetWorker\nsr\logs\daemon.log
If you have any questions, see article: NetWorker: How to use nsr_render_log.
  1. To disable this debugging, delete the vmbackup_logging file from the /nsr/tmp dir.

Additional Information

Related articles:

Affected Products

NetWorker

Products

NetWorker, NetWorker Series