BACKUPS FAILING

The goal of this discussion is to provide a central place for ALL BACKUP FAILURES/ISSUES/QUESTIONS.

This is an important and broad topic, so I expect this to be a work in progress and look forward to contributions from all.

Backups can fail for many reasons so it is useful to try to narrow things down at first by asking questions, such as:

• What is the backup environment? (vCenter VM image backups? / HyperV? / Win or Lin FS? / all?)
• Are ALL backups failing? (This would indicate a problem on the IDPA/Avamar/Data Domain side.)
• If ALL backups are failing- how does storage capacity look? (av GSAN for metadata and DD for data)
• Are only IDPA Internal backups failing or only customer backups failing?
• Are only some of the above mentioned types of clients failing? (This would indicate an issue on the customer vCenter or HyperV or Win/Lin client side.)
• What plugins are being used for the failed clients? (DDBoost? SQL? Win and/or Lin VMware Image? Win FS? Win VSS? etc) 
• Are all clients using that plugin failing?
• How many avamar proxies do you have in your environment? (if not enough can cause performance issues)
• Are all failures limited to a certain av proxy?
• Can you find anything in common with the failed clients that is different from the successful clients?
• Has anything changed in your environment that may have triggered the failures? (upgrades? changed passwords? etc)

One of the first places to look for clues is in the avamar aui- activity screen.

Select a failed client and select View logs as seen below. You can easily eyeball the logs here or you can Download logs. If the av aui screen won't come up, see the ssh cmds lower in this discussion to see it services are not running.

If only INTERNAL IDPA backups are failing, @willman_Perez has already posted a discussion for that here:

https://www.dell.com/community/Integrated-Data-Protection/Internal-Backups-failing/m-p/8100635#M93

If ALL backups are failing and/or you suspect a problem on the IDPA server side, you can ssh to avamar as admin and run these commands:

status.dpn
dpnctl status
mcserver.sh --version
free -m
grep -i heap /usr/local/avamar/var/mc/server_data/prefs/mcserver.xml
avmaint cpstatus
avmaint hfscheckstatus
avmaint gcstatus
cplist --lscp
dumpmaintlogs --types=gc --days=5 | grep -i "<4"
dumpmaintlogs --types=hfscheck --days=5 | grep -i "<4"
dumpmaintlogs --types=cp --days=5 | grep -i "<4"
mccli event show --unack=true
ddrmaint cplist

If you open a Support case, we would like to have the above information and cmd outputs and also an IDPA Support Bundle (acm dashboard, button in upper right corner "Create...", when it is finished it will let you know then at the same button choose "Download..")

So that is a good start. Please jump in with your contributions/questions etc and I will also be adding more over time.

If you found this useful please give KUDOS to help others find the information also.

thanks