The advanced setting SunRPC.MaxConnPerIP defines the maximum number of unique TCP connections that can be opened for a given IP address. This is of particular interest to users of NFS. If the number of mounts to an IP address is more than SunRPC.MaxConnPerIP, then the existing connections for NFS mounts are shared with new mounts from the same IP address. Currently VMware supports a maximum of 128 unique TCP connections per ESXi host but also supports up to 256 mounts per host. So what options are available to configure ESXi hosts to allow the maximum number of NFS mounts?
Take the following example. There are 4 IP addresses from which mounts are exposed from an NFS host or NAS array, and SunRPC.MaxConnPerIP is set to its default value of 4 (vSphere 5.1). This will give 4*4 = 16 unique TCP connections. After that, mounts of NFS datastores from those IP addresses will share the connection.
By setting the value to 32, this will give 4*32=128 unique TCP connections. However, if each IP address is exporting 64 mounts, then the connection to these 64 volumes will be shared among the 32 TCP connections defined in SunRPC.MaxConnPerIP. With this setting we can have 4*64=256 volumes. This means that each connection will share at least two mounts in this scenario.
So why would one need to be concerned about shared connections you might ask? For the most part, you do not. One caveat with sharing TCP connections however is that when one of the NFS volumes on a shared connection runs out of space, other NFS volumes that share the same connection might also report no space errors even though they still have adequate space. You can consider bumping the SunRPC.MaxConnPerIP value to its maximum value of 128 to avoid connection sharing. This issue and workaround are described in the ESXi 5.0 Update 2 Release Notes.
On another note, if you currently have tuned your host and already have 128 unique TCP connections consumed by NFS mounts and you attempt to mount an NFS datastore from a new IP address, the operation will not be successful and will fail with ‘Error while adding NFS mount: NFS connection limit reached!’ However, mounts of NFS datastores from already existing IP addresses will be successful as they can share connections.
Get notification of these blogs postings and more VMware Storage information by following me on Twitter: @CormacJHogan