Avamar: GC Failing with MSG_ERR_NOTAVAILABLE
Summary: Avamar Garbage Collection (GC) fails with the error "MSG_ERR_NOTAVAILABLE" due to stale sessions left from a node replacement or restart.
Symptoms
Avamar GC starts and then rapidly fails with MSG_ERR_NOTAVAILABLE.
Cause
Stale sessions were left from a node replacement or restart.
These stale backup, replication, or recovery sessions can cause Avamar GC to fail with MSG_ERR_NOTAVAILABLE.
Note that the "status.dpn" output shows 22 dispatchers (Dis column).
admin@avamar:~/>: status.dpn: Node IP Address Version State Runlevel Srvr+Root+User Dis Suspend Load UsedMB Errlen %Full Percent Full and Stripe Status by Disk 0.7 192.168.255.9 7.5.1-101_HF310405 ONLINE fullaccess mhpu+0hpu+0hpu 4 false 0.15 31582 34980635 5.4% 5%(onl:3098) 5%(onl:3076) 5%(onl:2980) 0.6 192.168.255.8 7.5.1-101_HF310405 ONLINE fullaccess mhpu+0hpu+0hpu 0 false 0.12 31583 24206783 5.4% 5%(onl:3055) 5%(onl:3021) 5%(onl:3074) 0.5 192.168.255.7 7.5.1-101_HF310405 ONLINE fullaccess mhpu+0hpu+0hpu 5 false 0.05 31578 32931941 5.4% 5%(onl:3018) 5%(onl:3051) 5%(onl:3059) 0.4 192.168.255.6 7.5.1-101_HF310405 ONLINE fullaccess mhpu+0hpu+0hpu 2 false 0.35 31590 30437344 5.2% 5%(onl:2983) 5%(onl:3033) 5%(onl:3035) 0.3 192.168.255.5 7.5.1-101_HF310405 ONLINE fullaccess mhpu+0hpu+0hpu 0 false 0.18 31574 14831616 5.4% 5%(onl:3102) 5%(onl:2994) 5%(onl:3043) 0.2 192.168.255.4 7.5.1-101_HF310405 ONLINE fullaccess mhpu+0hpu+0hpu 0 false 0.46 31575 13085565 5.2% 5%(onl:3014) 5%(onl:3033) 5%(onl:3024) 0.1 192.168.255.3 7.5.1-101_HF310405 ONLINE fullaccess mhpu+0hpu+0hpu 5 false 0.11 31581 36028694 5.3% 5%(onl:3046) 5%(onl:3055) 5%(onl:2990) 0.0 192.168.255.2 7.5.1-101_HF310405 ONLINE fullaccess mhpu+0hpu+0hpu 6 false 0.04 31585 34078136 5.3% 5%(onl:3029) 5%(onl:3063) 5%(onl:3028)
Confirm that there are no active backup or replication sessions with the "mccli" and "avmaint" commands
admin@avamar:~/>: mccli activity show --active 0,23000,CLI command completed successfully. ID Status Error Code Start Time Elapsed End Time Type Progress Bytes New Bytes Client Domain -- ------ ---------- ---------- ------- -------- ---- -------------- --------- ------ ------ admin@avamar:~/>: admin@avamar:~/>: avmaint sess | grep path admin@avamar:~/>:
Resolution
This particular issue started when a node was rebooted due to ping-no-ssh situation.
Multiple dispatchers show in the "status.dpn" output, but no corresponding activities are shown by "avmaint sess" or "mccli activity show."
Avamar GC performs a check at startup. The stale sessions interferes with this check.
Confirm with the backup team that it is acceptable to terminate all sessions. Stop the sessions through the GUI or command line.
Once all sessions have been terminated, use the following commands to recycle the maint and restart GC.
dpnctl stop maint
dpnctl start maint
GC should then start a new attempt and process successfully.