現象
During the upgrade, the cluster becomes inaccessible to end-users. The cluster indicates multiple nodes down and splits due to Read-Only (RO) status. Logs indicate that the nodes can no longer communicate with the iDRAC so they are being set RO. Filesystem operations may be suspended or aborted due to deadlocks, timeouts, or loss of quorum.
原因
During the upgrade process, if the iDRAC firmware version is being updated, the iDRAC reboots as part of the process. There is an approximately three minute window where the iDRAC is inaccessible. This prevents hardware health monitoring, so the node is set Read-Only until the iDRAC comes back online. The upgrade engine tracks the upgrade status in each neighborhood by disk pool reservations. When the node is set to RO during the iDRAC reboot, this cancels its existing disk pool reservation. Once the reservation is released, the upgrade engine initiates the upgrade on the next node in the neighborhood. This causes a progressively larger number of nodes to start upgrading until the cluster becomes wedged.
解決方法
If you have run into this issue, contact
Dell Technical Support and quote this knowledge base article ID.
The permanent fix for this issue is in the current version of OneFS. This document
does not apply if you are running:
- OneFS 9.4.0.19 and above
- OneFS 9.5.0.9 and above
- OneFS 9.7.1.0 and above
- OneFS 9.8.0.1 and above
For earlier versions of OneFS, clusters with Gen6.5 hardware, only use the --rolling switch when performing any upgrade that includes the Node Firmware Package. The parallel upgrade limitation is specific to when you are applying firmware. Command examples are:
# isi upgrade cluster start --parallel /ifs/data/OneFS_v9.5.0.7_LTS2023_V2_Install.isi (safe command as it does not include the node firmware package)
# isi upgrade firmware start --rolling --fw-pkg=/ifs/data/Isi_Fw_Package_v12.0.tar (safe command as you are doing a rolling upgrade)
# isi upgrade firmware start --parallel --fw-pkg=/ifs/data/Isi_Fw_Package_v12.0.tar (could induce DU as you are doing a firmware update with the parallel option)
# isi upgrade cluster start --parallel /ifs/data/OneFS_v9.5.0.7_LTS2023_V2_Install.isi --fw-pkg=/ifs/data/Isi_Fw_Package_v12.0.tar (could induce DU as you are including firmware in a parallel upgrade)
対象製品
PowerScale F200, PowerScale F600, PowerScale F900