IDPA models DP5x00 or DP8x00 with three ESXi hosts shows vSAN object inaccessible from vSphere UI and ESXi command line after restore from power outage. Rebooting the three ESXi hosts also does not help.
Here is an example:
- Show vSAN health status from any of the ESXi hosts:
# esxcli vsan health cluster list
Health Test Name Status
-------------------------------------------------- ------------------------
Overall health red (vSAN Object health)
Data red
vSAN object health red
Unable to display VM objects. It fails to turn on VM from vSphere web UI or command line with similar errors:
# vim-cmd vmsvc/getallvms
Skipping invalid VM '41'
Skipping invalid VM '43'
Skipping invalid VM '44'
Show vSAN cluster information found the maintenance mode is still set to ON from one of the cluster nodes:
# esxcli vsan cluster get
Cluster Information
Enabled: true
Current Local Time: 2021-04-11T00:44:48Z
Local Node UUID: 5afb4fd2-6e89-df35-0ab3-3cfdfe2caa40
Local Node Type: NORMAL
Local Node State: MASTER
Local Node Health State: HEALTHY
Sub-Cluster Master UUID: 5afb4fd2-6e89-df35-0ab3-3cfdfe2caa40
Sub-Cluster Backup UUID: 5afb4f4f-589e-acf6-1c85-3cfdfe2c89e0
Sub-Cluster UUID: 52740a1b-3621-f158-7b09-439048860c07
Sub-Cluster Membership Entry Revision: 2
Sub-Cluster Member Count: 3
Sub-Cluster Member UUIDs: 5afb4fd2-6e89-df35-0ab3-3cfdfe2caa40, 5afb4f4f-589e-acf6-1c85-3cfdfe2c89e0, 5afb4ef8-b1b7-1f4a-f923-3cfdfe2c88a0
Sub-Cluster Membership UUID: 4c137260-c6b9-0aa9-e52b-3cfdfe2caa40
Unicast Mode Enabled: true
Maintenance Mode State: ON
Config Generation: 66541957-e1a1-478b-9a87-106bcf5cb77a 25 2021-03-18T20:31:02.389
From the same ESXi host, when run show system maintenanceMode status, it says it is Disabled. This is inconsistent with above command output.
# esxcli system maintenanceMode get
Disabled
- SSH login to the ESXi host that shows inconsistent maintenance Mode status. Manually place the host into maintenance mode then move it out of maintenance mode:
# esxcli system maintenanceMode set --enable=true
# esxcli system maintenanceMode set --enable=false
- Check vSAN cluster status again. Expect to see the maintenance mode status show as OFF and the VMs can turn on without error:
# esxcli vsan cluster get
Cluster Information
Enabled: true
Current Local Time: 2021-04-11T00:44:48Z
Local Node UUID: 5afb4fd2-6e89-df35-0ab3-3cfdfe2caa40
Local Node Type: NORMAL
Local Node State: MASTER
Local Node Health State: HEALTHY
Sub-Cluster Master UUID: 5afb4fd2-6e89-df35-0ab3-3cfdfe2caa40
Sub-Cluster Backup UUID: 5afb4f4f-589e-acf6-1c85-3cfdfe2c89e0
Sub-Cluster UUID: 52740a1b-3621-f158-7b09-439048860c07
Sub-Cluster Membership Entry Revision: 2
Sub-Cluster Member Count: 3
Sub-Cluster Member UUIDs: 5afb4fd2-6e89-df35-0ab3-3cfdfe2caa40, 5afb4f4f-589e-acf6-1c85-3cfdfe2c89e0, 5afb4ef8-b1b7-1f4a-f923-3cfdfe2c88a0
Sub-Cluster Membership UUID: 4c137260-c6b9-0aa9-e52b-3cfdfe2caa40
Unicast Mode Enabled: true
Maintenance Mode State: OFF