VSAN Part 34 – how many disks are needed for stripe width

by Posted on

Yesterday I posted an article which discussed some common misconceptions with Virtual SAN that come up time and again. Once I published that article, I immediately had an additional question about basic VSAN behaviour and functionality related to stripe width. More specifically, the question is how many disks do you need to satisfy a stripe width requirement. Let’s run through it here.

Last year, I posted an article that described how many hosts you need in a VSAN Cluster to tolerate a certain number of failures. This is a similar sort of question, but it deals with VSAN’s ability to stripe a virtual machine object (e.g. VMDK) across multiple disks. The correct name for this capability is number of disk stripes per object and the maximum number that is supported is 12. So how may disks do I need to satisfy a stripe width of 12?

Let’s ask ourselves what putting a stripe width of 12 in the policy actually does. For every mirror/replica, the cluster will need to have at least this many magnetic disks (12) to stripe the mirror/replica across.

There is perhaps a misconception that physical disks can be used to hold multiple stripes; this is not the case when a stripe width is specified in the policy.

One other item to draw to your attention is the fact that the stripe width value is a minimum. VSAN may actually decide to create more stripes than what is placed in the policy, should it need to.

Example 1: StripeWidth=12, NumberOfFailuresToTolerate (FTT) = 0

In this example, FTT is set to 0, so there is no requirement to create mirror copies/replicas of the objects. Therefore, to satisfy this stripe width = 12 requirement, a total of 12 physical spindles will be needed in the cluster. So 3 of the disks could be on host 1, 3 of the disks could be on host 2, and the remaining 6 could be on host 3, and VSAN can happily deploy this VMDK object. Of course, if all 12 spindles are available on host 1, we should be good too.

Example 2: StripeWidth=12, NumberOfFailuresToTolerate (FTT) = 1

This policy requirement becomes a little trickier because now there is also a requirement to tolerate a failure. That means that if a host goes down, there must still be enough hosts and disks in the cluster to maintain a full copy of the data. So if this is a 3 node cluster, each node will ideally have 12 or more disks. This allows one replica to be placed on host1, and the second replica on host2, and a witness on host3.

If however each host has less than 12 disks, there will need to be additional hosts in the cluster to have all the stripes from one replica on a subset of hosts and the stripes from the other replica on a completely different subset of hosts. We cannot allow the components from both replicas to be placed on the same host, because if that host failed, it would impact both mirrors and leave us with an inaccessible object.

Example 3: StripeWidth=12, NumberOfFailuresToTolerate (FTT) = 1

This is another interesting example to discuss. Lets assume again that this is a 3 node cluster, and each node has an SSD and 3 magnetic disks. The policy requests a stripe width of 12. Will VSAN now begin to place multiple stripes on the same physical spindle? The answer is no. You will not be able to deploy a virtual machine which has this policy on this cluster as there are not enough resources. VSAN will tell you this; the deployment of a VM should fail with a message similar to “Need 12 disks but only found 3”.

Example 4: StripeWidth=4, NumberOfFailuresToTolerate (FTT) = 1

Let’s take the same example as before; 3 node cluster, each node with an SSD and 3 magnetic disks. Now there are 9 disks in the cluster, so it should be able to satisfy my stripe width requirement of 4 and tolerate 1 failure, right? No, incorrect. If VSAN allowed this policy to be deployed, it would end up having to place components from both replicas on the same host. For example:

  • Replica 1, Stripe 1a – Host 1 Disk 1
  • Replica 1, Stripe 1b – Host 1 Disk 2
  • Replica 1, Stripe 1c – Host 1 Disk 3
  • Replica 1, Stripe 1d – Host 2 Disk 1
  • Replica 2, Stripe 2a – Host 2 Disk 2
  • Replica 2, Stripe 2b – Host 3 Disk 1
  • Replica 2, Stripe 2c – Host 3 Disk 2
  • Replica 2, Stripe 2d – Host 3 Disk 3

As we can see, both replicas have had one stripe placed on the same host, host 2. Now if host 2 fails, we impact both replicas, so this configuration cannot tolerate a failure. This is why VSAN cannot deploy such a configuration. Again, the deployment of a VM should fail, reporting that there are not enough disks found.

Example 5: StripeWidth=3, NumberOfFailuresToTolerate (FTT) = 1

One final example using our 3 node cluster, each node with an SSD and 3 magnetic disks. This policy requirement will deploy just fine as there are enough physical disks to satisfy the stripe width (3 disks on one host), the replica copy (another 3 disks on a different host) and any witnesses required.

Multiple stripes on the same physical disk?

Now lets return to the misconceptions outlined earlier. Why would there be a misconception that VSAN can place multiple stripes on the same physical disk? Well, in fact VSAN will do this on certain occasions. I wrote about the reasons in a previous post in fact. But there is a distinction to be made. If the VM Storage Policy contains a stripe width requirement, this number of physical disks must be available. In this case stripes must be placed on different physical disks.

If however VSAN needs to stripe (chunks might be a better description) an object because of space constraints (physical disks too small, VMDK object greater than 255GB), then in this case VSAN can place multiple stripes/chunks on the same physical disk.

I hope this helps to explain the stripe width requirement if you plan to use it with VSAN.

Published by Cormac