VAAI-NAS – Some snapshot chains are deeper than others

by Posted on

This is something that was recently brought to my attention, and I wasn’t aware of this difference in behavior between the various storage vendors who implement VAAI-NAS. VAAI-NAS implements a number of different offload primitives, but the one we are interested in here is the Fast File Clone primitive which is the ability to offload the creation of snapshots/clones to the NAS storage array. This mechanism is also referred to as Native Snapshots. However, some arrays cannot support a full chain of snapshots.

When a VM snapshot is taken with VAAI-NAS using the Fast File Clone primitive, the storage array (using native snapshot or clone technology) will create an object that will then be presented back to the ESX host and appear as a VM Snapshot. Given that this would be cloning at the file level, each clone would be independent per VM, and would not occur at the datastore level like we do with a normal volume snapshot.

Now, it appears that different vendors support different levels of snapshots. For example, if I create a Virtual Machine disk file, and snapshot it, that snapshot create operation will be sent to the storage array if VAAI-NAS primitives are in place. However, my next snapshot may or may not be offloaded, depending on the implementation by the storage vendor. If the snapshot operation exceeds the allowed chain depth, a message ‘The number of snapshot levels has exceeded supported limits‘ is displayed in the logs.

For example, EMC’s EMCNasPlugin 1.0-10 version for VNX only goes to a single level before reporting that it has exceeded the supported limits. [Update: It appears that a 2012 release of VNX firmware includes a flag for chained snapshot support. This must be set when creating the volume however; it cannot be set post volume creation. This allows the level to increase to 2 snapshots in a chain. See comment section for further details]. NetApp’s NAS plugin can go much deeper; internally, I’ve seen it go to a depth of 10 snapshots, but NetApp claim that it can go much deeper than that.  In conversation with folks at HDS, it seems that they can accommodate much larger depths of snapshots on their HDS-HNAS product too.  At the time of writing, I wasn’t able to get an exact figure from the EMC Isilon folks. However, considering we only support a snapshot chain of 32 for virtual machine snapshots, then you may consider that NetApp & HDS VAAI-NAS implementations both meet this. I’ll try to confirm the EMC Isilon figure at a later date. [Update: Isilon has unlimited snapshot support. See comments.]

What happens when the supported snapshot chain depth is reached? In that case, we simply revert back to using host based snapshot technology and create snapshots using the vmfsSparse (redo log) format.

Published by Cormac

10 Replies to “VAAI-NAS – Some snapshot chains are deeper than others”

  1. Just a small hint regarding the VNX snapshot implementation. You need to consider if you want to use the snapshot offload function when you create the filesystem. There you can check a small box, but it is not possible (afaik) to add it later once your datastore is already in use.

  2. Actually the flag Bas is refering to enables the snap of snap or chain of snaps. The single VAAI-NAS snap is always active as far as I know.

    1. Sebastian,

      Would it be true to say that without this flag, you can take at most 1 depth of snapshot.

      But with the flag, you can actually offload a chain on them? Do you know how many you can offload if the flag is on? I think this is news to a lot of folks, as I’ve heard many folks run into the 1 snapshot offload limitation with VAAI-NAS on VNX.

      1. yes, since VNX Inyo Release in 2012 and with activated flag (the flag is called “VMware VAAI nested clone support”) on the file system you can have an additional layer, so: source -> snap -> snap

        If you do not set the flag the limit is one layer, meaning: source -> snap

        For clones (they are also offloaded using VAAI) there are no limits to number or depth of chains in any way (with and without flag)

        1. Thanks for this Sebastian. I think this is what has caught a lot of folks out. I’ll update the article with a reference to this comment.

  4. Hey Cormac. I can’t find an instance in the available documentation that confirms this authoritatively, but I DID spend some time walking the Isilon VAAI development team through the specific scenario you describe above. The response I got from them is that EMC Isilon allows an unlimited number of VAAI-enabled snapshots per VM.

  5. Pingback: Unchain My VM, You Worry Me Night & Day(Snapshots)

Comments are closed.