VAAI-NAS, vCloud Director and Cloning Offload Strangeness

I was in a conversation with one of my pals over at Tintri last week (Fintan), and he observed some strange behaviour when provisioning VMs from a catalog in vCloud Director (vCD). When he disabled Fast Provisioning, he expected that provisioning further VMs from the catalog would still be offloaded via the VAAI-NAS plugin. All the ESXi hosts have the VAAI-NAS plugin from Tintri installed. However,  it seems that the provisioning/cloning operation was not being offloaded to the array, and the ESXi hosts resources were being used for the operation instead. Deployments of VMs from the catalogs were taking minutes rather than seconds. What was going on?

If you are not familiar with Fast Provisioning, I wrote an article on the vSphere storage blog around this some time back. In a nutshell, Fast Provisioning allows vCD to use linked-clones for the provisioning of new vApps/VMs. Without Fast Provisioning enabled, VMs are provisioned as full clones. As the name suggests, the provisioning of linked clones is much fast than full clones.

During the investigation, it was found that if the catalog where the VM that was being provisioned from was previously used for Fast Provisioning, then it failed to offload the clone operation when Fast Provisioning was subsequently disabled.

As part of the testing, a new catalog was created and Fast Provisioning was not enabled.  Now when a VM is provisioned from this catalog, the full clone operation is offloaded to the array, and the provisioning is almost instantaneous.

So what was the difference between having a catalog where Fast Provisioning was enabled, then disabled, and a catalog where fast provisioning was never enabled? It appears to be the following: when Fast Provisioning is enabled, a snapshot of the VMs in the catalog is taken but is not  removed when Fast Provisioning is disabled. This seems to be the crux of the issue. By clearing down this snapshot (consolidation via vCenter seems to do it), the offloading of full clones to the array when a VM is provisioned from the catalog is once again possible

Bottom line – if you have a vCD catalog that is used for Fast Provisioning, and you decide to disable it, full clones of VMs during provisioning are not offloaded. To offload full clones, you will need to consolidate any snapshots that are left on the VM from the Fast Provisioning setup, or a new catalog (without fast provisioning enabled) would need to be created for the provisioning operation to be offloaded to the array.

Also, although this was found by the guys over at Tintri, I suspect this is not a Tintri specific issue but may be encountered with other VAAI-NAS implementation. If you are using VAAI-NAS from other storage partners, and also use vCD, I’d be interested in hearing from you if you can verify this behaviour.