Unsolved
This post is more than 5 years old
29 Posts
0
5165
flow control with iSCSI + Nexus switches
We used to configure "flowcontrol receive on" on the nexus ports connecting our arrays and hosts.... The latest version of the Host Configuration guide mentions that flow control should be disabled....
Just wanted to confirm what are the latest best practices.... Also, it looks like support for DCB is not yet in place?
Thanks - Didier.
Below is an example of one of the host port as seen on the Nexus 5596 side
!Command: show running-config interface Ethernet186/1/23
!Time: Thu Apr 14 22:35:31 2016
version 6.0(2)N2(3)
interface Ethernet186/1/23
description coe-xen607g_iscsi-a
switchport access vlan 3897
flowcontrol receive on
vlab-k10-n5k-3-top# show int eth186/1/23
Ethernet186/1/23 is up
Hardware: 1000/10000 Ethernet, address: 3c5e.c3b1.0958 (bia 3c5e.c3b1.0958)
Description: coe-xen607g_iscsi-a
MTU 1500 bytes, BW 10000000 Kbit, DLY 10 usec
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA
Port mode is access
full-duplex, 10 Gb/s
Beacon is turned off
Input flow-control is on, output flow-control is on
Switchport monitor is off
EtherType is 0x8100
Last link flapped 00:06:34
Last clearing of "show interface" counters 32w1d
40 interface resets
30 seconds input rate 1068008 bits/sec, 140 packets/sec
30 seconds output rate 534112 bits/sec, 95 packets/sec
Load-Interval #2: 5 minute (300 seconds)
input rate 1.75 Mbps, 330 pps; output rate 10.17 Mbps, 912 pps
RX
25432589144 unicast packets 75 multicast packets 16 broadcast packets
25432589238 input packets 18728312514391 bytes
0 jumbo packets 0 storm suppression bytes
3 runts 0 giants 0 CRC 0 no buffer
3 input error 0 short frame 0 overrun 0 underrun 0 ignored
0 watchdog 0 bad etype drop 0 bad proto drop 0 if down drop
0 input with dribble 0 input discard
0 Rx pause
TX
69662597238 unicast packets 22800361 multicast packets 3537905 broadcast packets
69688936113 output packets 99317457830451 bytes
54 jumbo packets
0 output error 0 collision 0 deferred 0 late collision
0 lost carrier 0 no carrier 0 babble 0 output discard
20180816 Tx pause
Kumar_A
727 Posts
0
April 19th, 2016 16:00
The reason we advise against flow control is that enabling flow control serializes the flow for iSCSI port; hence causing major performance impact. It does not make sense to buy XtremIO and then enable flow control which limits the performance.
dcontis
29 Posts
0
April 20th, 2016 20:00
Thanks… Just wanted to double-check is that this is a departure from an older (early 2014 / code 2.x) recommendation ->
Enable flow control on all network switch ports(so on Cisco switches, this would be link level flow control) that handle iSCSI traffic as well as any host based adapters. If the host is using a software iSCSI initiator and NIC combination or HBA to handle iSCSI traffic, flow control must be enabled on the NICs and HBAs to obtain the performance benefit. Flow control ensures a receiver can make the sender pace its speed and is important in avoiding data loss.
Explanation: Typically in iSCSI SAN configurations, many servers(initiators) are communicating with the 10G storage ports on the XtremIO nodes. This can sometimes lead to an imbalance in the network traffic between the initiators that send network traffic and the ports that receive the traffic. If senders transmit data simultaneously, they may exceed the throughput capacity of the receiver. When this occurs, the receiver may drop packets, forcing senders to retransmit the data after a delay. Although this will not result in any loss of data, latency will increase because of the retransmissions of dropped packets, and I/O performance will degrade. Flow control allows the receiver to instruct the sender to pause transmission of additional data when the receiver senses that it is being overwhelmed. The receiver does this by sending pause frames to the sender, which causes the sender to stop packet transmission for a short period of time. This pause allows the receiver to process its backlog so it can later resume accepting input. According to Cisco, while latency is introduced into the SAN by flow control, it is much smaller when using flow control than when flow control is disabled and packets must be retransmitted.