I will start with a caveat. The plan is to support both Solid State Disks and PCIe flash devices on VSAN. However, for the purposes of this post, I will refer to this flash resource as an SSD for simplicity.
SSDs serve two purposes in VSAN. They act as both a read cache and a write buffer. This dramatically improves the performance of virtual machines running on the vsanDatastore. In some respects VSAN can be compared to a number of ‘hybrid’ storage solutions in the market, which also use a combination of SSD & HDD to boost the performance of the I/O, but which have the ability to scale out capacity based on low-cost HDD. Read on to learn more about read cache and write buffering on VSAN.
The read cache keeps a list of commonly accessed disk blocks. This reduces the I/O read latency in the event of a cache hit, i.e. the disk block is in cache. What is interesting in the context of VSAN is that the actual block that is being read by the application running in the virtual machine may not be on the same ESXi host on which the virtual machine is running. In other words, the VM’s compute may be on one ESXi host and its storage objects may be on a different ESXi host. In this case, the VSAN directory service is referenced to find whether the block may be in the read cache of another ESXi host in the cluster. If there is a cache miss, then the data is retrieved directly from the HDD.
VSAN always tries to make sure that it sends a given read to the same replica so that the block gets cached only once in the cluster. Cache space is expensive, so this mechanism optimizes how much cache your VM consumes.
The write cache behaves as a non-volatile write buffer. The writes from an application in a VM are acknowledged when they land on the SSD. The fact that we can use SSD for writes also reduces the latency for write operations.
Because the writes go to SSD, we must of course ensure that there is copy of the data elsewhere in the VSAN cluster in the event of a failure. All virtual machines deployed to VSAN have an availability policy setting that ensures at least one additional copy of the virtual machine data is available. This includes the write cache contents. Once a write is initiated by the application running inside of the Guest OS, the write is forked to both the write cache on the host which is the ‘owner’ of the storage object, and also to the write cache on the remote host or hosts which contain a replica.
This means that in the event of a host failure, we also have a copy of the in-cache data on another host in the system so no loss of data will happen; the virtual machines will simply reuse the replicated copy of the cache as well as the replicated disk data.
At regular intervals, the SSD write buffer is de-staged to HDD.
As with many other flash solutions, sizing your VSAN SSD configuration so that your virtual machines ‘working set’ resides completely in cache is going to give the best possible performance.
Check out all my VSAN posts here.