PowerScale: Cluster Becomes Inaccessible During a Parallel Firmware Upgrade

Summary: During a parallel OneFS upgrade that includes the "--fw-pkg" switch, or a parallel firmware only upgrade containing PowerScale nodes (F200, F210, F600, F710, F900, or F910), the cluster becomes inaccessible. ...

This article applies to This article does not apply to This article is not tied to any specific product. Not all product versions are identified in this article.
Check out other resources

Symptoms

During the upgrade, the cluster becomes inaccessible to end-users. The cluster indicates multiple nodes down and splits due to Read-Only (RO) status. Logs indicate that the nodes can no longer communicate with the iDRAC so they are being set RO. Filesystem operations may be suspended or aborted due to deadlocks, timeouts, or loss of quorum.

Cause

During the upgrade process, if the iDRAC firmware version is being updated, the iDRAC reboots as part of the process. There is an approximately three minute window where the iDRAC is inaccessible. This prevents hardware health monitoring, so the node is set Read-Only until the iDRAC comes back online. The upgrade engine tracks the upgrade status in each neighborhood by disk pool reservations. When the node is set to RO during the iDRAC reboot, this cancels its existing disk pool reservation. Once the reservation is released, the upgrade engine initiates the upgrade on the next node in the neighborhood. This causes a progressively larger number of nodes to start upgrading until the cluster becomes wedged.

Resolution

If you have run into this issue, contact Dell Technical Support and quote this knowledge base article ID.

The permanent fix for this issue is in the current version of OneFS. This document does not apply if you are running:
  • OneFS 9.4.0.19 and above
  • OneFS 9.5.0.9 and above
  • OneFS 9.7.1.0 and above
  • OneFS 9.8.0.1 and above
For earlier versions of OneFS, clusters with Gen6.5 hardware, only use the --rolling switch when performing any upgrade that includes the Node Firmware Package. The parallel upgrade limitation is specific to when you are applying firmware. Command examples are: 
# isi upgrade cluster start --parallel /ifs/data/OneFS_v9.5.0.7_LTS2023_V2_Install.isi (safe command as it does not include the node firmware package)

# isi upgrade firmware start --rolling --fw-pkg=/ifs/data/Isi_Fw_Package_v12.0.tar (safe command as you are doing a rolling upgrade)

# isi upgrade firmware start --parallel --fw-pkg=/ifs/data/Isi_Fw_Package_v12.0.tar (could induce DU as you are doing a firmware update with the parallel option)

# isi upgrade cluster start --parallel /ifs/data/OneFS_v9.5.0.7_LTS2023_V2_Install.isi --fw-pkg=/ifs/data/Isi_Fw_Package_v12.0.tar (could induce DU as you are including firmware in a parallel upgrade)

Affected Products

PowerScale F200, PowerScale F600, PowerScale F900
Article Properties
Article Number: 000222012
Article Type: Solution
Last Modified: 06 Feb 2026
Version:  7
Find answers to your questions from other Dell users
Support Services
Check if your device is covered by Support Services.