The GSAN was recently rolled back to a validated checkpoint, and now the MCS will not start. When tailing the tmp dpnctl mcs start output log, an error stating the gsan rollbacktime does not match stored rollbacktime was observed.
This can apply when NetWorker VMware Protection integration is configured with the Virtual Backup Appliance (VBA) or the Integrated Data Protection Appliance (IDPA) or any Avamar Server.
root@avamar:~/#: dpnctl status
Identity added: /home/dpn/.ssh/dpnid (/home/dpn/.ssh/dpnid)
dpnctl: INFO: gsan status: up
dpnctl: INFO: MCS status: down.
dpnctl: INFO: Backup scheduler status: down.
dpnctl: INFO: axionfs status: down.
dpnctl: INFO: Maintenance windows scheduler status: enabled.
dpnctl: INFO: Unattended startup status: enabled.
dpnctl: INFO: [see log file "/usr/local/avamar/var/log/dpnctl.log"]
root@avamar:~/#: tail -f /tmp/dpnctl-mcs-start-output-15545
Database server is running...
INFO: Starting messaging service.
INFO: Started messaging service.
=== BEGIN === check.mcs (prestart)
check.mcs passed
=== PASS === check.mcs PASSED OVERALL (prestart)
Starting Administrator Server...
schedule_version key has value: 7.1.0.1, expecting value: 7.1.1.1.
The Administrator Server preferences file needs to be updated.
ERROR: gsan rollbacktime: 1495040451 does not match stored rollbacktime: 1477305907
The timestamp on the MCS and GSAN are off since the GSAN was rolled back to a previous point in time.
- Force an MCS restore so the MCS and GSAN are synced.
root@labvba01:~/#: dpnctl start mcs --force_mcs_restore
Identity added: /home/dpn/.ssh/dpnid (/home/dpn/.ssh/dpnid)
dpnctl: INFO: Restoring MCS data...
dpnctl: INFO: MCS data restored.
dpnctl: INFO: Starting MCS...
dpnctl: INFO: To monitor progress, run in another window: tail -f /tmp/dpnctl-mcs-start-output-16158
dpnctl: WARNING: 1 warning seen in output of "[ -r /etc/profile ] && . /etc/profile ; /usr/local/avamar/bin/mcserver.sh --start"
dpnctl: INFO: MCS started.
root@labvba01:~/#: dpnctl status
Identity added: /home/dpn/.ssh/dpnid (/home/dpn/.ssh/dpnid)
dpnctl: INFO: gsan status: up
dpnctl: INFO: MCS status: up.
dpnctl: INFO: Backup scheduler status: down.
dpnctl: INFO: axionfs status: down.
dpnctl: INFO: Maintenance windows scheduler status: enabled.
dpnctl: INFO: Unattended startup status: enabled.
dpnctl: INFO: [see log file "/usr/local/avamar/var/log/dpnctl.log"]
- MCS is restored, now start backup scheduler.
root@avamar:~/#: dpnctl start sched
Identity added: /home/dpn/.ssh/dpnid (/home/dpn/.ssh/dpnid)
dpnctl: INFO: Resuming backup scheduler...
dpnctl: INFO: Backup scheduler resumed.
- Validate the GSAN, MCS, and backup scheduler are up.
root@labvba01:~/#: dpnctl status
Identity added: /home/dpn/.ssh/dpnid (/home/dpn/.ssh/dpnid)
dpnctl: INFO: gsan status: up
dpnctl: INFO: MCS status: up.
dpnctl: INFO: Backup scheduler status: up.
dpnctl: INFO: axionfs status: down.
dpnctl: INFO: Maintenance windows scheduler status: enabled.
dpnctl: INFO: Unattended startup status: enabled.
dpnctl: INFO: [see log file "/usr/local/avamar/var/log/dpnctl.log"]