I was discussing this issue with a good friend of mine over at Tintri, Fintan Comyns. Fintan was seeing some strange behaviour with the cloning on Windows 2008 R2 Guest OS running in virtual machines using the Tintri VAAI-NAS plugin, and wanted to know if this behaviour was normal or not. Basically what he was seeing was that a clone operation of a virtual machine was not being offloaded. Rather, he was seeing two separate independent snapshots (snapshots that were not in a chain, but both pointing to the base VMDK) were getting created at the time of the cloning operation. Fintan also reported that if they used to the Sync Driver or stopped VMware Tools altogether in the Windows 2008 R2 Guest OS, the operation worked and the clone operation was being offloaded. The same operation was tried with a Windows 7 Guest OS running in a virtual machine, and in this case a single snapshot was created which was offloaded. so what was going on? It had us scratching our heads for a while.
A couple of additional details:
- ESXi 5.5
- vCenter 5.5
- VAAI-NAS plugin vmware-esx-TintriVaaiNasPlugin v220.127.116.11-2.7
Unfortunately we do not have VAAI-NAS statistics in esxtop. There are statistics for the block primitives, but not the NAS ones. However you can figure out if a VAAI-NAS clone operation is being offloaded or not. From the network statistics on esxtop, you should be able to tell if a clone operation is sending writes over the network. A clone operation which is being offloaded to the array by VAAI-NAS should not be writing over the network to the destination.
Since this investigation was done on a Tintri array, we can use their very nice UI to help with the observations of VAAI-NAS behaviour. The UI displays the Windows 2008R2 source disk, a couple of clones which offloaded successfully and a clone which did not. The “Used GiB” column highlights that both the VM that was offloaded by VAAI-NAS and the VM that was cloned with the Tintri UI are space efficient. But the VM that was not offloaded with VAAI-NAS has its clone copy of the data written to disk, so it takes up the same written space as the source initially. On a Tintri array, a cloned VM normally takes up zero additional space. The mouse pointer has been left to hover over the “16.5” value for the source Windows 2008R2 VM to show its breakdown of space used.
- This seemed specific to Windows 2008. It did not affect other flavours of Windows.
- It only occurred when VSS (also known as Volume Snapshot Service or the Volume Shadow Copy Service) was enabled. If VSS was disabled as per KB article 1031298, or the VMware Tools sync driver was used or VMware Tools was disabled, the cloning operation offloaded.
- If the disk type is changed to dynamic within the Guest OS, the clone also offloaded.
The answer to this behaviour was eventually found in the following VMware document:
To quote that document:
Windows 2008 application level quiescing is performed using a hardware snapshot provider. After quiescing the virtual machine, the hardware snapshot provider creates two redo logs per disk: one for the live virtual machine writes and another for the VSS and writers in the guest to modify the disks after the snapshot operation as part of the quiescing operations. The snapshot configuration information reports this second redo log as part of the snapshot. This redo log represented the quiesced state of all the applications in the guest. This redo log must be opened for backup with VDDK 1.2. The older VDDK 1.1 software cannot open the second redo log for backup.
So it seems that hot clones of virtual machines running Windows 2008 will not offload via VAAI-NAS due to the application level quiescing. This is not unique to Tintri; you will also observe this with other VAAI-NAS plugins. Kudos to Fintan for observing this behavior in the first place, and coming up with the root cause.