Why is uploading files to VMFS so slow?
This is something which has come up numerous times, and behavior which many of you have observed. There seems to be some issue with uploading files to a VMFS datastore. In fact, in one example, we had someone report that it took 10 minutes to upload a Windows 7 ISO to an iSCSI datastore and less than 1 minute to upload the same ISO to an NFS datastore. Both datastores were very healthy and fast, and both had running VMs on them. There have been variations of this behavior reported before. This post will try to explain why.
Copying files to VMFS is slower than NFS due to the metadata operational overheads. Copying files piece-by-piece results in a lot of metadata updates as the file grows. As a file grows in little chunks, VMFS will have to continually invoke metadata operations to allocate new resources on the VMFS. Much of this would require the disk to be locked to ensure data integrity. NFS doesn’t need to do that as it has a very different file locking mechanism. This is why we (VMware) always recommend using vmkfstools to move virtual machines between datastores at the command line.
One way of verifying this behaviour is to look at the VAAI statistics for VMFS datastores backed by VAAI capable storage arrays, or SCSI reservation requests/reservations per second for the non-VAAI datastores. A simple example is to gzip a very large VMDK on a VMFS volume which sorts of does the same things (lots of very small reads and writes). What you will observe is a very large number of locks occurring. Even with VAAI, we observed via esxtop a very large number of ATS and ZERO requests taking place on the volume:
ATS ATSF ZERO ZERO_F MBZERO/s
42052 3 61370 0 5.72
From this, we can surmise that the gzip operation locks the resource/block, zeroes it out, and then copies the new contents in. The same would hold through for a file upload operation to VMFS either via the UI, or a CLI copy command.
Hopefully this sheds some light on why these types of operations are slower on VMFS than they are on NFS.
Thanks Cormac for the quick and simple explanation.
So, this also relates to restore activity by a backup software using the management network for example?
Luca.
Indeed – this could well be the case. Many of the vendors can do hot-add of disks to leverage in-kernel tools for copying – which should avoid this. However, if you compare a network restore versus a disk-to-disk restore, you should be able to check on the number of metadata operations involved.
thanks, this is in fact for example what Veeam suggests about having at least one VM configured in order to leverage hotadd for restores, and now thanks to this post I also know the technical reason.
Luca.
Cormac, can you talk about what specific metadata is being updated as the file grows?
-Josh
Hi Josh, sure. As we grow the file, we allocate another cluster of resources from the VMFS (blocks). We then have to update the file’s metadata (think inode) to reflect its new size. Since VMFS is clustered, we don’t want anyone other host to try to allocate this block of resources or update the metadata of the file at the same time, so locks are needed when we do operations like this.