NFS Best Practices – Part 4: Sizing Considerations

Welcome to the next installment of NFS Best Practices. In this fourth and final best practice section on NFS, I asked a number of our major NAS storage partners some sizing questions. The questions basically fell into 3 different categories:

  1. Do you have a recommended NFS volume size/sweet-spot recommendation?
  2. Do you have a volume block size recommendation that should be configured on the array?
  3. Do you have a recommended number of VMs per NFS datastore?

In fact, the responses from the vendors were all pretty similar. Let’s take a look at what they had to say.

Maximum volume Size/Sweet Spot

Many of you will have observed the following statement in the VMware Configuration Maximums GuideContact your storage array vendor or NFS server vendor for information about the maximum NFS volume size. The purpose of this first question was to see if there was a volume size which worked well which the partners agreed on. In fact, all partners responded saying that there was no performance gain or degradation depending on the volume size – customers could build NFS volumes of any size, so long as it was below the array vendor’s supported maximum. This can be up in the hundreds of TB, but consensus appears to be that the majority of NFS datastores are in the 10s of TB in size. The datastores sizes vary quite a bit from customer to customer.

Sizing of volumes is typically proportional to the number of VMs you wish to deploy, plus snapshots/changed blocks for backup purposes. Another consideration is the fact that many arrays now have deduplication and compression features, which will also reduce capacity requirements.

Recommended Block Size

From the vendors that I spoke to, this parameter is not tuneable for the most part. Some vendors have it hard set to 4KB and others have it hard set to 8KB. Block sizes are typically a multiple of 4KB. These align nicely with the 4KB grain size used in VMware’s virtual disk format (VMDK). For those vendors who have it set to 8KB, the recommendation is to format the volumes in the Guest OS to a matching 8KB block size for optimal performance. The one vendor who did have a tuneable block size had different recommendations depending on the workload – 4KB for random workloads and 32KB for sequential workloads. Since having multiple virtual machines sharing an NFS datastore is going to result in random workloads, you probably would be using 4KB. In this area, it is best to speak to your storage array vendor to get vendor specific advice. The questions one should ask are:

  1. What is the volume block size on the array?
  2. Is it tuneable?
  3. If so, what should I tune it to? (Be sure to explain that the datastore will be used by VMs, so the workload will be random for the most part)
  4. Are there any considerations when formatting the volumes in the Guest OS?

RSIZE & WSIZE

I thought it useful just to add a note about the RSize and WSize parameters when discussing sizing. These parameters, defined at mount time, define the I/O chunk transfer sizes between the host and the target. These sizes are not tuneable. However, feedback from the partners suggest that the current size of 64KB meets their requirements. While some storage arrays can now support much larger Wsize & Rsize values (up to 1MB), there is still no way to change these settings in vSphere at the time of writing.

 Recommended number of VMs per NFS datastore

This was again met with a pretty consistent response from all vendors. The number of virtual machines which can run on a single datastore is directly proportional to the infrastructure and the workloads running in the VMs, i.e. hundreds of low I/O VMs versus a few very intensive I/O VMs. Network congestion was an important factor, which brings us back to some recomendations which were made in part 1 of this series which dealt with network best practices for NFS. Since VMware still only supports NFS v3 over TCP/IP, one recommendation was to use more datastores presented over different interfaces so that traffic could be balanced over multiple TCP sessions.

The other major factor is related to the backup and recovery SLA. If you have one single datastore with lots of VMs, how long are you willing to wait while this is restored in the event of a failure. This is starting to become the major consideration in the ‘how many VMs per datastore’ debate.

Another consideration which a customer highlighted is the snapshot technology used by the backup product – is it using array based snapshots or is it using virtual machine snapshots. Performance considerations are needed if using virtual machine snapshots to concurrently capture point in time copies of VMs. In many cases, array based snapshots have less impact on the datastores and are more scalable when it comes to backups. There may be some array based limitations to take into considerations too – for instance, the number of snapshot copies of a VM that a customer wishes to maintain versus the number of snapshot copies an array can support. This would vary from vendor to vendor so best to check this configuration maximum with your storage array vendor.

From a vSphere snapshot perspective, refer to this KB article 1015180 for further details around snapshots and their usage. From KB article 1025279, VMs can support up to 32 snapshots in a chain, but VMware recommends that you use only 2-3 snapshots in a chain, and to use no single snapshot for more than 24-72 hours.

Links to previous NFS Best Practices parts

Get notification of these blogs postings and more VMware Storage information by following me on Twitter: @CormacJHogan

7 thoughts on “NFS Best Practices – Part 4: Sizing Considerations

  1. Supporting pNFS in ESX would be really great. Especially for modern scale-out storage systems.

    • Hey Markus,
      It think we’ve had this conversation before, right? :-)
      I bring it up at every opportunity I get internally – we’ll all just have to wait and see.

  2. Hi Cormac,

    Great series on NFS! I used a lot of this for setting up my brand-new Oracle Sun ZFS Appliances behind our vSphere 5.1 infrastructure.

    Got some advice from another Oracle customer to use a “Database record size” (the term equates to ZFS volume block sizes) of 128k because ZFS is smart enough to do smaller IOs when needed. Bit of a deviation from Part 4 of your NFS Best Practices but might be specific to ZFS. Any thoughts on that? Here’s a thread I started on the VMware Communities regarding these Sun Oracle ZFS Appliances:
    http://communities.vmware.com/thread/433290

    Thanks for all the great info that you’ve shared!

    -Nate

    • I haven’t come across this Nate. Is the 128k block size a documented best practice from Sun/Oracle? The white paper which I will be publishing shortly on NFS Best Practices will detail some of the common block sizes recommended by array vendors, but I do say that one should contact the NFS array/appliance/server vendor for their own specific recommendations.

      • Hi Cormac,

        Unfortunately, there isn’t a lot of explicit guidance from Oracle around NFS on ZFS. There’s a fair bit of published info on the block size of iSCSI LUNs that are going to be formatted for VMFS but in all the NFS literature, they appear to recommend a default Database Record Size of 128k except for databases where they go with 8k. Their most recent guidance (September 2012) is focused on vSphere 4.x and was not as detailed as I might have liked.

        ZFS uses variable size blocks (see http://en.wikipedia.org/wiki/ZFS#Variable_block_sizes). Perhaps using a larger block size with NFS gives ZFS the opportunity to choose the way the IO is handled? Just taking a guess on this.

        Best,
        Nate

  3. Pingback: Welcome to vSphere-land! » Storage Links