Dell VxRail Architecture Overview

Network and latency

Both L2 and L3 support VMware vSAN connectivity between the data nodes. L2 does not require a static route, but L3 requires a static route.

The following table provides a description of the intersite communication for L2 and L3:

Table 1. Intersite communicationThe following table provides a description of the intersite communication for L2 and L3:
Inter-site	Deployment scenario	Layer	Routing
Site-to-site	Default	L2	Not required.
Site-to-site	Default	L3	Static routes are required.
Site-to-witness	Default	L3	Static routes are required.
	Default	L3	Static routes are required when using an interface other than the management (vmk0) interface.
	Witness traffic separation	L2 for 2-node	Static routes are not required.

Figure 1. Network layer connectivity between VMware vSAN nodes

Static routes are required because VMware vSAN uses the default TCP stack. A VMkernel specific gateway is not supported. The VMkernel management interface uses the default gateway that is also used by any other VMkernel interface using the same TCP stack. Isolate the VMware vSAN network from normal infrastructure traffic on a dedicated network. The VMware vSAN network cannot use the default gateway and static routes must be used to connect with L3 addresses. While used for site-witness communication over L3, it can also include site-site communication when using L3 for data node addresses.

Use L3 for connectivity between the data sites and the witness. Witness Traffic Separation (WTS) can be used with stretched cluster configurations under the following conditions:

If a VMkernel interface other than the management VMkernel interface is tagged with witness traffic, static routes are required to communicate with the VMware vSAN witness host VMkernel interface tagged for VMware vSAN traffic.
If the management VMkernel interface is tagged with witness traffic, static routes are not required if the host can already communicate with the VMware vSAN witness host VMkernel interface using the default gateway.

The MTU can be different. For data nodes, the MTU can be 9000 and between data nodes and the witness, the MTU can be 1500.

Witness traffic separation

The witness traffic separation offers the following features:

An alternate VMkernel interface can be designated to carry traffic that is destined for the witness rather than the VMware vSAN tagged VMkernel interface.
More flexible network configurations are separated by allowing separate networks for node-to-node and node-to-witness traffic.
Two independent subnets and routes can be advertised from each data node site to the witness site for routing purposes.
You can set up a VxRail 2-node cluster.

Supported geographical distances

For VMware vSAN stretched clusters, support is based on network latency and bandwidth requirements instead of distance. The key requirement is the actual latency numbers between sites.

Bandwidth

The bandwidth requirement between the main sites depends on the workload on VMware vSAN, the amount of data, and the handling of failure scenarios.

The following table describes the basic bandwidth and latency requirements under normal operating conditions:

Table 2. Basic bandwidth and latency requirementsBasic bandwidth and latency requirements
Site	Bandwidth	Latency
Site-to-site	VMware vSAN OSA: Minimum of 10 Gbps VMware vSAN ESA: Minimum of 10 GbE.	<5 millisecond latency RTT
Site-to-witness	2 Mbps per 1000 VMware vSAN components	< 500 millisecond latency RTT for 1 host per site <200 millisecond latency RTT (up to 10 hosts per site) <500 millisecond latency RTT (1 host per site)

Site-to-witness network latency

In most VMware vSAN stretched-cluster configurations, latency or RTT between sites hosting VM objects and the witness nodes should not be greater than 200 msec (100 millisecond one way).

The latency to the witness depends on the number of objects in the cluster. VMware recommends that on VMware vSAN stretched cluster configurations up to 10+10+1, a latency of less than or equal to 200 milliseconds is acceptable. If possible, a latency of less than or equal to 100 milliseconds is preferred. For configurations that are greater than 10+10+1, VMware requires a latency of less than or equal to 100 milliseconds.

For site-to-witness calculation examples, see Appendix B: Bandwidth calculation examples.

Intersite MTU

Unless witness traffic separation is used, maintain a consistent MTU between data nodes and the witness in a stretched cluster configuration.

Ensure that each VMkernel interface designated for VMware vSAN traffic is set to the same MTU to prevent traffic fragmentation. The VMware vSAN health check looks for a uniform MTU across the VMware vSAN data network, and reports on any inconsistencies.

Connectivity

The following connectivity options are available:

The management network provides connectivity to all three sites.
VM network connectivity between the data sites (the witness does not run VMs that are deployed on the VMware vSAN cluster).
VMware vSphere vMotion network connectivity between the data sites (VMs are never migrated from a data host to the witness host).
VMware vSAN network connectivity to all three sites.
VMware vSAN data traffic on the ISL.
VMware vSAN metadata traffic on the uplinks to the witness.

Welcome

Welcome to Dell