PowerFlex3.x ESXi SDC Loses Connectivity after MDM Failover

Scenario
After MDM failover, PowerFlex (3.5/3.6) SDCs lose connectivity to the backend storage and the current Primary MDM.

Symptoms
 - The backend (MDM/SDS) was installed/upgraded to 3.5 or above.
 - The frontend (SDC) was installed/upgraded to 3.5 or above.
 - The MDMs have four data IPs and no Virtual IPs in use.
 - This is a 5-node MDM cluster.
 - The /etc/vmware/esx.conf has all 12 MDM IPs (3 MDMs x 4 IPs) included in the "IoctlMdmIPStr" parameter.
 - The MDM IPs in the /etc/vmware/esx.conf are all IPs from one MDM first, then the next MDM, and then a third MDM.
    Example: All IPs ending in .31 are MDM1, .32 is MDM2 and .33 is MDM3.

/vmkernel/module/scini/options = "IoctlIniGuidStr=D3EE6457-9535-444F-B0DD-778D57D74C7B IoctlMdmIPStr=10.1.1.31,10.1.2.31,10.1.3.31,10.1.4.31,10.1.1.32,10.1.2.32,10.1.3.32,10.1.4.32,10.1.1.33,10.1.2.33,10.1.3.33,10.1.4.33"

 - SDCs only lose connectivity when MDM3 is the Primary MDM in this example. If MDM1 or MDM2 is Primary MDM, the SDCs operate as normal.
 - The vmkernel.log from the ESXi host shows this on boot when parsing the "IoctlMdmIPStr" parameter. Note no IPs are ending in .33 that are successfully parsed:

2021-11-07T16:24:04.482Z cpu18:2098300)WARNING: PowerFlex ioctl_HandleAddMdmFromModuleParam:610 :Received 10.1.1.31,10.1.2.31,10.1.3.31,10.1.4.31,10.1.1.32,10.1.2.32,10.1.3.32,10.1.4.32,10.1.1.33,10.1.2.33,10.1$
2021-11-07T16:24:04.482Z cpu18:2098300)WARNING: PowerFlex Ioctl_ParseIpStr:576 :Succefully Parsed IP[1] 10.1.1.31 for MDM.
2021-11-07T16:24:04.482Z cpu18:2098300)WARNING: PowerFlex Ioctl_ParseIpStr:576 :Succefully Parsed IP[2] 10.1.2.31 for MDM.
2021-11-07T16:24:04.482Z cpu18:2098300)WARNING: PowerFlex Ioctl_ParseIpStr:576 :Succefully Parsed IP[3] 10.1.3.31 for MDM.
2021-11-07T16:24:04.482Z cpu18:2098300)WARNING: PowerFlex Ioctl_ParseIpStr:576 :Succefully Parsed IP[4] 10.1.4.31 for MDM.
2021-11-07T16:24:04.482Z cpu18:2098300)WARNING: PowerFlex Ioctl_ParseIpStr:576 :Succefully Parsed IP[5] 10.1.1.32 for MDM.
2021-11-07T16:24:04.482Z cpu18:2098300)WARNING: PowerFlex Ioctl_ParseIpStr:576 :Succefully Parsed IP[6] 10.1.2.32 for MDM.
2021-11-07T16:24:04.482Z cpu18:2098300)WARNING: PowerFlex Ioctl_ParseIpStr:576 :Succefully Parsed IP[7] 10.1.3.32 for MDM.
2021-11-07T16:24:04.482Z cpu18:2098300)WARNING: PowerFlex Ioctl_ParseIpStr:576 :Succefully Parsed IP[8] 10.1.4.32 for MDM.
2021-11-07T16:24:04.483Z cpu18:2098300)WARNING: PowerFlex ioctl_HandleAddMdmFromModuleParam:681 :Create finish for MDM. Number of IP addresses 8. Status (SUCCESS)
2021-11-07T16:24:04.483Z cpu18:2098300)WARNING: PowerFlex ioctl_HandleAddMdmFromModuleParam:696 :Total Received 1 MDMs Cluster with total 8 IPs

- The PowerFlex WEBUI will state that "The SDC is either down or unreachable over the network" for any SDC in this scenario.

 Impact

Data are unavailable for any SDCs in this scenario.

PowerFlex 3.5 has changed the number of IPs that are allowed in the /etc/vmware/esx.conf "IoctlMdmIPStr" parameter to eight MDM IPs. As the SDC only knows about the first 8 IPs, if the primary MDM ever fails over to MDM3 in this scenario it cannot contact and use those IPs, and thus lose connectivity to the storage.

For an immediate fix, change the Primary MDM back to MDM1 or MDM2.

Another workaround is to change the order of MDM IPs in the "IoctlMdmIPStr" location, so all three MDMs have IPs in the first 8 allowable IPs. This will require a reboot of the ESXi host.

The permanent fix can be found in later PowerFlex code

Impacted Versions

PowerFlex 3.5 and 3.6

Fixed In Version

PowerFlex v3.6.0.4 and above

PowerFlex 3.5.1.6 and above