3 Posts
0
977
Grid Storage Nodes went offline
Hello, need a little help, we have a drive in a node go offline and when it did it made the node go offline, we have replaced the drive but the node is still offline. During this time we lost power to two other nodes and now the grid is in an "readonly" state. Is there any way to get the nodes back online and bring the grid back into a "normal" state for use again? The nodes have power and are online (for the two that lost power). Grid is 7.5.1 with a mix of gen4 and gen4s nodes.
ionthegeek
2K Posts
0
November 16th, 2020 12:00
Any system where multiple nodes have gone offline concurrently will need to be rolled back to the most recently created checkpoint. This can be done using the "dpnctl" utility.
ETTech
3 Posts
0
November 16th, 2020 12:00
would you mind if I asked what the steps are for that? Or a run down of it?
ETTech
3 Posts
0
November 16th, 2020 15:00
I was able to use the --help and figured out what I needed to do, however I have this now, and this is the node that was offline, upon review the idrac6 in it has crashed and is not recoverable, anyway around it? Last I check the system was booting but was stuck at start ipmi drive.
dpnctl: INFO: Checking that gsan was shut down cleanly...
dpnctl: ERROR: problem running command "export SYSPROBEUSER=admin && /usr/local/avamar/bin/mapall --noerror --givestatus 'export PATH=.:~admin:/usr/local/avamar/bin:$PATH ; (gsan isclean ; echo $? >/tmp/dpnctl-gsan-isclean-status-9930) >/tmp/dpnctl-gsan-isclean-output-9930 2>&1' >/tmp/dpnctl-mapall-output-9930 2>&1" - exit status 1
dpnctl: ERROR: missing gsan "isclean" status information from node "0.4"
dpnctl: ERROR: unable to determine "isclean" status of gsan - some nodes did not report, or else did not report as expected
dpnctl: ERROR: traceback on exit:
main::gsan_is_clean (/usr/local/avamar/bin/dpnctl line 2062)
main::gsan_restart (/usr/local/avamar/bin/dpnctl line 3176)
main::dpn_up (/usr/local/avamar/bin/dpnctl line 5151)
main::handle_top_level_command (/usr/local/avamar/bin/dpnctl line 6198)
dpnctl: ERROR: [user "admin"] program exit status = 1 (error)