There are no other network changes that occurred on this cluster,
- Node is showing network partitioned, although it can ping other nodes via vmKping
- Node is in vSAN cluster partition.
- VSAN health page showing that host3 is disconnected from VC, it is actually failing API calls
localcli vsan cluster get
Cluster Information:
Enabled: true
Current Local Time: 2019-03-03T07:40:03Z
Local Node UUID: 5b90382f-a7d7-8dc2-4358-246e96e1013a
Local Node Type: NORMAL
Local Node State: MASTER
Local Node Health State: HEALTHY
Sub-Cluster Master UUID: 5b90382f-a7d7-8dc2-4358-246e96e1013a
Sub-Cluster Backup UUID:
Sub-Cluster UUID: 5292b573-6d0c-d8e2-b233-aa15b43a3740
Sub-Cluster Membership Entry Revision: 0
Sub-Cluster Member Count: 1
Sub-Cluster Member UUIDs: 5b90382f-a7d7-8dc2-4358-246e96e1013a
Sub-Cluster Membership UUID: aa767b5c-5cc5-a904-8224-246e96e1013a
Unicast Mode Enabled: true
Maintenance Mode State: OFF
Config Generation: 2a3be2d8-0006-4b0f-a87a-887ed8242d79 3 2019-03-03T06:39:42.986
[root@xxxxx:~]
++ We are able to ping vSAN nodes using vmkping command.
[root@xxxx:/tmp] esxcfg-vmknic -l | grep -i vmk3
vmk3 20336 IPv4 172.x.x.x 255.xx.2xx.0 172.x.x.x 00:50:56:66:aa:ed 1500 65535 true STATIC defaultTcpipStack
vmk3 20336 IPv6 xxxx::xxxx:xxxx:xxxx:xxxx 64 00:50:56:66:aa:ed 1500 65535 true STATIC, PREFERRED defaultTcpipStack
[root@xxxx:/tmp]
[root@xxxx:~] vmkping -I vmk3 172.x.x.x
PING 172.16.200.10 (172.16.200.10): 56 data bytes
64 bytes from 172.x.x.x : icmp_seq=0 ttl=64 time=6.226 ms
64 bytes from 172.x.x.x : icmp_seq=1 ttl=64 time=2.879 ms
64 bytes from 172.x.x.x : icmp_seq=2 ttl=64 time=0.134 ms
--- 172.x.x.x ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 0.134/3.080/6.226 ms
[root@xxxxx:~] vmkping -I vmk3 172.xx.xx.11
PING 172.x.x.x (172.x.x.x ): 56 data bytes
64 bytes from 172.x.x.x: icmp_seq=0 ttl=64 time=0.200 ms
64 bytes from 172.x.x.x: icmp_seq=1 ttl=64 time=0.163 ms
64 bytes from 172.x.x.x: icmp_seq=2 ttl=64 time=0.152 ms
--- 172.x.x.x ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 0.152/0.172/0.200 ms
Looks like unicast registration is problem here. Unicastagent list was missing on two nodes
Add the missing unicast agent entries
esxcli vsan cluster unicastagent add -t node -u <uuid> -U <unicastSupport> -a <ip> -p <port> -i <interface>
Then
Example:
esxcli vsan cluster unicastagent add -t node -u 57ea62b0-9df6-6532-110c-9457a55a37d0 -U true -a XX.XXX.XXX.XX-p 12321 -i vmk3
[root@xxxxx:~] esxcli vsan cluster unicastagent list
NodeUuid IsWitness Supports Unicast IP Address Port Iface Name
------------------------------------ --------- ---------------- ------------- ----- ----------
5b90378f-0b0b-2da4-933b-246e96e101b6 0 true 172.x.x.x 12321
5b90380e-927b-73ce-159f-246e96e1018a 0 true 172.x.x.x 12321
[root@xxxx:~]
[root@xxxxxx:~] esxcli vsan cluster unicastagent list
NodeUuid IsWitness Supports Unicast IP Address Port Iface Name
------------------------------------ --------- ---------------- ------------- ----- ----------
5b90378f-0b0b-2da4-933b-246e96e101b6 0 true 172.x.x.x 12321
5b90382f-a7d7-8dc2-4358-246e96e1013a 0 true 172.x.x.x 12321
[root@xxxxx:~]
[root@xxxx:/tmp] esxcli vsan cluster unicastagent list
NodeUuid IsWitness Supports Unicast IP Address Port Iface Name
------------------------------------ --------- ---------------- ------------- ----- ----------
5b90380e-927b-73ce-159f-246e96e1018a 0 true 172.x.x.x 12321
5b90382f-a7d7-8dc2-4358-246e96e1013a 0 true 172.x.x.x 12321
[root@xxxxx:/tmp]
[root@xxxx:~] esxcli vsan cluster get
Cluster Information
Enabled: true
Current Local Time: 2019-03-03T07:48:55Z
Local Node UUID: 5b90382f-a7d7-8dc2-4358-246e96e1013a
Local Node Type: NORMAL
Local Node State: AGENT
Local Node Health State: HEALTHY
Sub-Cluster Master UUID: 5b90378f-0b0b-2da4-933b-246e96e101b6
Sub-Cluster Backup UUID: 5b90380e-927b-73ce-159f-246e96e1018a
Sub-Cluster UUID: 5292b573-6d0c-d8e2-b233-aa15b43a3740
Sub-Cluster Membership Entry Revision: 3
Sub-Cluster Member Count: 3
Sub-Cluster Member UUIDs: 5b90378f-0b0b-2da4-933b-246e96e101b6, 5b90380e-927b-73ce-159f-246e96e1018a, 5b90382f-a7d7-8dc2-4358-246e96e1013a
Sub-Cluster Membership UUID: d519d05b-a227-5621-66dc-246e96e101b6
Unicast Mode Enabled: true
Maintenance Mode State: OFF
Config Generation: 2a3be2d8-0006-4b0f-a87a-887ed8242d79 3 2019-03-03T06:39:42.986
This content is translated in other languages:
| https://downloads.dell.com/TranslatedPDF/PT-BR_KB530995.pdf |
| https://downloads.dell.com/TranslatedPDF/ZH-CN_KB530995.pdf |
| https://downloads.dell.com/TranslatedPDF/AR-EG_KB530995.pdf |
| https://downloads.dell.com/TranslatedPDF/DE_KB530995.pdf |
| https://downloads.dell.com/TranslatedPDF/ES_KB530995.pdf |
| https://downloads.dell.com/TranslatedPDF/FR_KB530995.pdf |
| https://downloads.dell.com/TranslatedPDF/KO_KB530995.pdf |
| https://downloads.dell.com/TranslatedPDF/JA_KB530995.pdf |
| https://downloads.dell.com/TranslatedPDF/IT_KB530995.pdf |
| https://downloads.dell.com/TranslatedPDF/NL_KB530995.pdf |
| https://downloads.dell.com/TranslatedPDF/PT_KB530995.pdf |
| https://downloads.dell.com/TranslatedPDF/RU_KB530995.pdf |
| https://downloads.dell.com/TranslatedPDF/SV_KB530995.pdf |