Mutex waits data domain backup for sql server

Question

Hello For a customer we are using Data Domain (Version 19.1) for executing SQL Server Backups (full and Transaction log). The backups are being executed via T-SQL Scripts. For most of the servers we are not facing any issues. However, we keep having problems with  transaction log backups of always on availability groups. In the data domain log there are mutex waits visible, which last for over 10 minutes. The backups completes successfully but takes between 15 to 20 minutes. Without this issue backups complete within 5-8 minutes.  What is strange is that this problem does not occur for all of the servers where Always on is being used and that it disappears after server reboot. However, it keeps coming back. This is the data domain log:   1/03/20 00:34:44 D:/views/nw/appagent_19_1/nsr/db_apps/bsmsql/sqlutil.cpp(15372): encoded index directory [MSSQL2%3aBEMS%2dEWS%2dCO].11/03/20 00:34:44 D:/views/nw/appagent_19_1/nsr/db_apps/bsmsql/sqlxbsacompforsave.cpp(89):BackupItem 0 allocated11/03/20 00:34:44 D:/views/nw/appagent_19_1/nsr/db_apps/bsmsql/sqlxbsacompforsave.cpp(911): Creating DB Locking Mutex object MSSQL%3aBEMS%2dEWS%2dCO%3aINCR11/03/20 00:34:54 Mutex wait timed out 1 th time.11/03/20 00:35:04 Mutex wait timed out 2 th time.11/03/20 00:35:14 Mutex wait timed out 3 th time.11/03/20 00:35:24 Mutex wait timed out 4 th time.11/03/20 00:35:34 Mutex wait timed out 5 th time.11/03/20 00:35:44 Mutex wait timed out 6 th time.11/03/20 00:35:54 Mutex wait timed out 7 th time.11/03/20 00:36:04 Mutex wait timed out 8 th time.11/03/20 00:36:14 Mutex wait timed out 9 th time.11/03/20 00:36:24 Mutex wait timed out 10 th time.11/03/20 00:36:34 Mutex wait timed out 11 th time.11/03/20 00:36:44 Mutex wait timed out 12 th time.11/03/20 00:36:54 Mutex wait timed out 13 th time.11/03/20 00:37:04 Mutex wait timed out 14 th time.11/03/20 00:37:14 Mutex wait timed out 15 th time.11/03/20 00:37:24 Mutex wait timed out 16 th time.11/03/20 00:37:34 Mutex wait timed out 17 th time.11/03/20 00:37:44 Mutex wait timed out 18 th time.11/03/20 00:37:54 Mutex wait timed out 19 th time.11/03/20 00:38:04 Mutex wait timed out 20 th time.11/03/20 00:38:14 Mutex wait timed out 21 th time.11/03/20 00:38:24 Mutex wait timed out 22 th time.11/03/20 00:38:34 Mutex wait timed out 23 th time.11/03/20 00:38:44 Mutex wait timed out 24 th time.11/03/20 00:38:54 Mutex wait timed out 25 th time.11/03/20 00:39:04 Mutex wait timed out 26 th time.11/03/20 00:39:14 Mutex wait timed out 27 th time.11/03/20 00:39:24 Mutex wait timed out 28 th time.11/03/20 00:39:34 Mutex wait timed out 29 th time.11/03/20 00:39:44 Mutex wait timed out 30 th time.11/03/20 00:39:54 Mutex wait timed out 31 th time.11/03/20 00:40:04 Mutex wait timed out 32 th time.11/03/20 00:40:14 Mutex wait timed out 33 th time.11/03/20 00:40:24 Mutex wait timed out 34 th time.11/03/20 00:40:34 Mutex wait timed out 35 th time.11/03/20 00:40:44 Mutex wait timed out 36 th time.11/03/20 00:40:54 Mutex wait timed out 37 th time.11/03/20 00:41:04 Mutex wait timed out 38 th time.11/03/20 00:41:14 Mutex wait timed out 39 th time.11/03/20 00:41:24 Mutex wait timed out 40 th time.11/03/20 00:41:34 Mutex wait timed out 41 th time.11/03/20 00:41:44 Mutex wait timed out 42 th time.11/03/20 00:41:54 Mutex wait timed out 43 th time.11/03/20 00:42:04 Mutex wait timed out 44 th time.11/03/20 00:42:14 Mutex wait timed out 45 th time.11/03/20 00:42:24 Mutex wait timed out 46 th time.11/03/20 00:42:34 Mutex wait timed out 47 th time.11/03/20 00:42:44 Mutex wait timed out 48 th time.11/03/20 00:42:54 Mutex wait timed out 49 th time.11/03/20 00:43:04 Mutex wait timed out 50 th time.11/03/20 00:43:14 Mutex wait timed out 51 th time.11/03/20 00:43:24 Mutex wait timed out 52 th time.11/03/20 00:43:34 Mutex wait timed out 53 th time.11/03/20 00:43:44 Mutex wait timed out 54 th time.11/03/20 00:43:54 Mutex wait timed out 55 th time.11/03/20 00:44:04 Mutex wait timed out 56 th time.11/03/20 00:44:14 Mutex wait timed out 57 th time.11/03/20 00:44:24 Mutex wait timed out 58 th time.11/03/20 00:44:34 Mutex wait timed out 59 th time.11/03/20 00:44:44 Mutex wait timed out 60 th time.11/03/20 00:44:54 Mutex wait timed out 61 th time.11/03/20 00:44:54 Waiting for more than 600 seconds, a potential deadlock. Anyways continue to create the database list...11/03/20 00:44:54 ReleaseMutex failed with error 288   Does anyone know, why this could be happening? Regards Martina

crazyrov · Answer

Hi @msordakis , Are you using NetWorker (NMM) for backup or DDBEA agent ?

msordakis · Answer

Hi @crazyrov  I am using DDBMA agent.

crazyrov · Answer

ok, Mutex generally used to break a deadlock condition, I am assuming cleaning was not running during this time. Can you check if you can use another SU can backup the DB's? I recommend you to open an SR with support to check if there is any issue on the DD that is causing this condition?

NetWorker

Was this post helpful?