Avamar: MC flushes are overdue due to MCS flushing locally
Summary: Avamar: MC flushes are overdue due to the Management Console Server (MCS) flushing locally.
Symptoms
The following alert appears in the Avamar Administrator UI:
Flush of Administrator Server data to server is overdue
The same event can be seen when using MCCLI:
mccli event show --unack
0,23000, CLI command completed successfully.
ID Date Type Code Category Severity Domain Summary
------ ----------------------- ----- ----- -------- -------- ------ --------------------------------------------------------
539105 2016-08-12 15:22:58 EDT ERROR 22407 SYSTEM PROCESS / Flush of Administrator Server data to server is overdue.
538595 2016-08-12 11:06:34 EDT ERROR 22407 SYSTEM PROCESS / Flush of Administrator Server data to server is overdue.
A manual flush is successful:
mcserver.sh --flush
=== BEGIN === check.mcs (preflush)
check.mcs passed
=== PASS === check.mcs PASSED OVERALL (preflush)
Flushing Administrator Server...
Administrator Server flushed.
No new MC_BACKUPS are created:
avmgr getb --path=/MC_BACKUPSCause
MCS is flushed locally because the nodelist cache is not able to update.
A typo within /usr/local/avamar/var/probe.xml is preventing MCS to process probe.xml correctly.
Resolution
1. Verify that pg_dump is successful:
cat /usr/local/avamar/var/mc/server_log/mcserver.log.0 |grep "pg_dumpall exited with rc"
The following event is seen if the pg_dump is successful:
INFO: ------pg_dumpall exited with rc=0------
2. Check if the flush is saving locally:
cat /usr/local/avamar/var/mc/server_log/mcserver.log.0 |grep "flushing locally"
The following event is seen if the flush is saving locally:
08/22-08:45:00.00005 com.avamar.mc.dpn.DPNProxyService.flush
WARNING: Flushing locally because DPN is Unknown State
If this event is not present, stop following this article.
If this event is present, go to step 3.
3. If the error is caused by an invalid probe.xml, the following entries are present in the mcserver logs:
08/22-08:45:45.00946 com.avamar.mc.util.MCException.logException
WARNING: java.lang.NullPointerException
at java.util.TreeMap.put(Unknown Source)
at java.util.TreeSet.add(Unknown Source)
at com.avamar.mc.cm.ProbeNetworkInterfaceInfo.<init>(ProbeNetworkInterfaceInfo.java:105)
at com.avamar.mc.cm.ProbeXmlNodeInfo.<init>(ProbeXmlNodeInfo.java:90)
at com.avamar.mc.cm.ProbeXmlInfo.<init>(ProbeXmlInfo.java:1578)
at com.avamar.mc.dpn.NodelistCache.update(NodelistCache.java:487)
at com.avamar.mc.dpn.PollerBase.refresh(PollerBase.java:98)
at com.avamar.mc.dpn.DPNPoller.run(DPNPoller.java:87)
at java.lang.Thread.run(Unknown Source)
The error above is seen if the "NodelistCache" thread is unable to process probe.xml.
Correct any typos in the probe.xml file.