NVP-vProxy Triage Guide: NetWorker VMware Protection vProxy Triage and Log Overview
Summary: This KB provides assistance in identifying logs on the vProxy and references for debug and VMware/vCenter log locations.
Instructions
GETTING STARTED
NVP-vProxy stands for NetWorker VMware Protection with the vProxy Appliance. This appliance is used to backup and restore virtual machines with NetWorker. Backup data is stored as VMDK files on the Data Domain target device. It can also be cloned to disk or tape devices.This article outlines how to approach initial analysis of an issue with the NVP-vProxy solution.
ISSUE DETAILS
In order to generate a complete problem description, address the following questions:- What operation is being attempted which is not working?
- Is the issue consistent or intermittent?
- If intermittent, is there a known trigger?
- Did this work better before?
- When did the issue first occur (and if known, what changed)?
- What is the precise expected behavior and what is this expectation based on?
- Times of occurrences and observed behavior trend
- Does the issue happen only at times of heavy load on the backup environment or backups of a large group of Virtual Machines?
- What is the scope of the issue (all policies/some policies, all VMs/some VMs)?
- What has been tried so far to fix and what conclusions have been drawn from this?
ENVIRONMENT DETAILS
- Full NetWorker server version/build and operating system platform.
- vProxy appliance version (In NMC under Devices->VMware Proxies).
- Overview of how many vProxies are in use.
- ESXi and vCenter versions are in use.
- Name of VM being backed up/recovered that is experiencing issues.
- Target media for these backups; including device name and DDOS version.
PROXY HEALTHCHECK
The ProxyHC tool was developed to help diagnose and correct some issues which may be observed when using the NVP vProxy appliance:
NVP-vProxy: How to use health check tool ProxyHC on vProxy appliance
LOG COLLECTION
NOTE: See "Additional Info" section for enabling debug options on NVP vProxy. For most initial diagnosis debug is not required; however, debug may be requested if standard logging is determined to be insufficient.
Log collection processes are detailed in: NVP-vProxy: Log Collection Processes for VMware Protection Issues
REGISTRATION ISSUES:
Start on the NetWorker server daemon.raw:
Linux: /nsr/logs/daemon.raw
Window: C:\Program Files\EMC NetWorker\logs\daemon.raw
NetWorker: How to use nsr_render_log
On the vProxy appliance: /opt/emc/vproxy/runtime/logs/vrapid/vrapid-engine.log
BACKUP ISSUES:
Start on the NetWorker server with the policy logs:
Linux: /nsr/logs/policy/policy_name
Window: C:\Program Files\EMC NetWorker\logs\policy\policy_name
vProxy Backup (log overview):
- /opt/emc/vproxy/runtime/logs/vbackupd
- /opt/emc/vproxy/runtime/logs/recycle/vbackupd/DATE (logs moved here after job has completed or failed)
| Log Name | Function |
| <session-uuid>.log | Backup Session Log |
| <daemon>-engine.log | vbackupd service engine log |
| <daemon>-snapmgr.log | VMware Snapshot Manager log |
| <daemon>-boost.log | DD Boost session log |
| <daemon>-vddk.log | VMware VDDK session log |
Troubleshooting Common Backup Errors:
- NetWorker: NVP vProxy Troubleshooting VM Backup and Restore Performance and Throughput
- NVP vProxy: VM backups failing with "no matching devices for save of client 'vCenter Name': check storage nodes, device or pools"
- NVP vProxy: (Troubleshooting Guide) Backups fail with "NSR VMware proxy is not available for datastores"
- NVP vProxy: VM backups fail with "VDDK Error 13: You do not have access rights to this file" - Troubleshooting/Triage Guide
- NVP vProxy: Troubleshooting VMware Snapshot Issues during NetWorker backups.
RECOVERY ISSUES:
On the NetWorker server:- /nsr/logs/recover/ Saved recover session logs from NMC recover wizard
- /nsr/logs/restapi/restapi.log RESTAPI log
- Linux: /nsr/authc/logs/flr-server.log FLR UI web-service
- Windows: \nsr\authc-server\tomcat\logs\flr-server.log FLR UI web-service
- /nsr/logs/adhoc/nsrvproxy_flr/<jobid>.log FLR troubleshooting
- /opt/emc/vproxy/runtime/logs/vrecoverd/
- /opt/emc/vproxy/runtime/logs/recycle/vrecoverd/DATE (logs moved here after job has completed or failed)
- /opt/emc/vproxy/runtime/logs/vflrd
- /opt/emc/vproxy/runtime/logs/recycle/vflrd/DAT (logs moved here after job has completed or failed)
| Log Name | Function |
| <daemon>-engine.log | vrecoverd or vflrd service log. |
| <daemon>-boost.log | DD Boost session log |
| <daemon>-vddk.log | VMware VDDK session log |
| mountsessionnumber.log | FLR Mount session log |
| browsesessionnumber.log | FLR Browse session log |
Target VM Logs:
Windows: C:\Program Files (x86)\EMC\vProxy FLR Agent\logs\vflragentd\vflragentd-agent.log
Linux:/opt/emc/vproxyra/logs/vflragentd/vflragentd-agent.log
MSVMAPPAGENT LOGS:
The following section provides location information for all logs associated with SQL application-consistent data protection.In order to increase the debug level for SQL application-consistent virtual machine (MSVMAPPAGENT) backups, use dbgcommand, For example, dbgcommand -p <nsrd-pid> Debug=9.
Once you complete the debugging session, ensure that you reset the debug level of nsrd to zero by running dbgcommand <nsrd-pid> Debug=0.
MSVMAPPAGENT (VM) logs:
You can access logs related to MSVMAPPAGENT from the following locations on the SQL VM:
- C:\Program Files\DPSAPPS\MSVMAPPAGENT\logs\msvmagent_discovery.log Discovery log.
- C:\Program Files\DPSAPPS\MSVMAPPAGENT\logs\msvmcatsnap.log FULL backup.
- C:\Program Files\DPSAPPS\MSVMAPPAGENT\logs\msvmagent_appbackup.log Transaction log backup.
- C:\Program Files\DPSAPPS\MSVMAPPAGENT\logs\msvmagent_snapshotrestore.log Restore of FULL backup.
- C:\Program Files\DPSAPPS\MSVMAPPAGENT\logs\msvmagent_apprestore.log Restore of Transaction log backup.