I was having some discussions recently on the community forums about Virtual SAN behaviour when a VM storage policy is changed on-the-fly. This is a really nice feature of Virtual SAN whereby requirements related to availability and performance can be changed dynamically without impacting the running virtual machine. I wrote about it in the blog post here. However there are some important considerations to take into account when changing a policy on the fly like this.
Changes to the policy of a VM (or an object) on VSAN can be made which do not impact the already existing objects of the VM. For example, increasing the number of failures to tolerate would add a new mirror/replica to the object, which would simply sync up to the already existing replica(s). Also, when the flash read cache reservation or proportional capacity is reduced, this does not trigger a new object creation or resync. VSAN will simply update the component policies to reflect the changes in these cases. Here is an example where Number Of Failures To Tolerate is increased from 0 to 1, leaving the original object(s) intact and requiring no component/object rebuild:
However there are other scenarios which would require a whole new set of components and replicas to to be created and synchronized. One example is changing a stripe width of an object. This involves a complete different layout of the data, and thus requires a new replica to be created and synchronized against the original object. When this is complete Virtual SAN will discard the original object. The nice thing about this is that it can take place without impacting the virtual machine, although some overhead in incurred when this rebuilding of the object takes place.
Scenarios where a new replica may need to be instantiated and resynced are as follows:
1. Enlarge a read cache reservation.
2. Change the stripe width.
3. Enlarge Object Space Reservation.
4. Change number of failures to tolerate when read cache reservation is not 0
Each of these instantiate a new replica, synchronizes it against the original replica and then discards the original. The thing to keep in mind with changing policies on-the-fly like this is that there needs to be enough capacity on the VSAN datastore to accommodate a temporary instantiation of the replica or replicas that are affected by this policy change. When dealing with virtual machines that have relatively small VMDKs, this isn’t too much of a concern and there really isn’t that much data to synchronize. However, for very large VMDKs, quite a significant amount of free space is needed, and the amount of data that needs to synchronize may also be a concern. Note that we are talking about the actual written data to the VMDK, and not the logical size of the VMDK. Here is an example of increasing the stripe width from 1 and 2, showing the a component/object rebuild required:
There is one exception to the case of increasing the stripe width. If the stripe width of an object is increased in the policy, but the object already as this number of stripes across physical disks, then a full resync is not needed since this object is already compliant. The reason that the object may already have the required number of stripes could be due to the fact that it is a very large object, and since the maximum size of a component is 255GB, it is already striped. Or it could be because VSAN has split the object into small chunks and distributed them to multiple disks because there is only a small amount of free disk space in the cluster (remember that stripe width requirement in the policy is treated as a minimum – VSAN may stripe more if necessary).
Another consideration is when an object is expanded, for example, when a VMDK is grown. When an object is expanded, the first step is to add a concatenation to the replica(s) to support the increased size requirement. This step shouldn’t trigger any resync. However, over a period of time, VSAN will create new replicas to support the new size to include the concatenation, and the original replicas and concatenation will be discarded. Note that this attempt to merge the original replica and concatenation into a new replica can fail if VSAN have no place to put the new components and that is OK. VSAN will simply keep retrying to merge the concatenation periodically. Once the resyncing is complete for the new components, the old components will be cleaned up.
One final factor that should be considered is the following. VSAN will begin automatically to re-balance components on a physical disk when that physical disk reaches an 80% full threshold. It will try to place components on other disks in the cluster to bring all disks below the 80% threshold. When there are many physical disks in an environment, with lots of small components and lots of free capacity on the remaining disks in a cluster, this is fairly seamless. The disk selection algorithm will try to spread the new components as fairly as possible to keep all the disks uniformly filled. However for other environments, (e.g. that have very large VMDKs but only a handful of physical disks), when a request is made to change the policy dynamically and this requires that a new, very large VMDK to be instantiated, this can lead to VSAN to not only instantiate this new VMDK, but if it pushes any physical disk over the 80% threshold, it may also start VSAN to auto-balance the components depending on the overall cluster state.
The advice here is that although changing policy on-the-fly is a great feature, ensure that there are enough available resources and capacity in the cluster before you use it. Since rebuilding of components/replica may also be involved, consider on-the-fly changing of policies as a maintenance task, does outside normal working hours.