NETAPP vs VMWARE FLOW CONTROL DILEMMA

vQuicky

> Performance improves were seen on environments with ESXi 5.1/NetApp/10G switches having

Flow control disabled

> VMware recommends leaving flow control enabled while NetApp best practice recommends disabling it if using 10G switches.

> VMware recommends investigating pause frames and if too many were found – indicates at an underlying problem.

 

inDepth

We had recently seen some random datastore drops and issues in our virtualized environment which had a backend Netapp storage. Upon investigation and some deep-dive it was found that flow control was enabled on the entire stack.

This is another article
worth looking – talks about NetApp sending too many pause frames.

Below is what Netapp has to say in its best practice guide about flow control.

Flow control is a low-level process for managing the rate of data transmission between two nodes to
prevent a fast sender from overrunning a slow receiver. Flow control can be configured on ESX/ESXi
servers, FAS storage arrays, and network switches. For modern network equipment, especially 10GbE
equipment, NetApp recommends turning off flow control and allowing congestion management to be
performed higher in the network stack. For older equipment, typically GbE with smaller buffers and
weaker buffer management, NetApp recommends configuring the endpoints, ESX servers, and NetApp
arrays with the flow control set to “send.”

So Netapp says for modern 10GbE equipment –  the flow control should be turned off and allow the congestion to be managed higher up in the stack. This does make sense now that applications are now smart enough to manage data throughput and flow. Hardware flow control is really not application aware unless you write you application to look out for that and that may not be good application technique.

On the flip side VMware actually asks to leave flow control enabled by default in the hypervisor. Below is what VMware says about flow control.

Note: By default, flow control is enabled on all network interfaces in VMware ESX and ESXi. This is the preferred configuration. If there are large numbers of pause frames in an environment, it is usually indicative of an underlying issue that should be investigated.

VMware advises on investigating pause frames on the hypervisor and if you were to see “too many” then there may be a issue lower in the stack. But there is no definite number of what “too many” really means. Below is what VMware says.

Pause Frames are related to Ethernet flow control and are used to manage the pacing of data transmission on a network segment. Sometimes, a sending node (ESX/ESXi host, switch, etc) may transmit data faster than another node can accept it. In this case, the overwhelmed network node can send pause frames back to the sender, pausing the transmission of traffic for a brief period of time.

If flow control is undesired in an environment, the support for flow control can be disabled for a given network interface or driver on the ESX/ESXi hosts. The method differs for different drivers.

So all in all – my recommendation is to have flow control disabled on your hosts, switches and even on the storage end. This can really help remove a lot of problems including dropping luns and NFS mounts.

To disable flow control on the hypervisor –

0 – No flow control.
1 – Flow control on received (RX) traffic only.
2 – Flow control on transmitted (TX) traffic only.
3 – Flow control on both received and transmitted traffic.

• To disable Flow Control for all ports of a quad-port Intel e1000 network interface:# esxcli system module parameters set –module e1000 –parameter-string “FlowControl=0”

Feel free to comment or correct me 🙂

The VMware KB article

The NETAPP best practices guide.