I’ve been involved in a few conversations recently regarding how VSAN trace files are handled when the ESXi host that is participating in a VSAN cluster boots from a flash device. I already did a post about some of these considerations in the past, but focused mostly on USB/SD. However SATADOM was not included in this discussion, as we did not initially support SATADOM in VSAN 5.5, and only announced SATADOM support for VSAN 6.0. It seems that there are some different behaviors that need to be taken into account between the various flash boot devices, which is why I decided to write this post.
Let’s start with ESXi hosts that are booting from either USB sticks, or SD cards. I’m grouping these together since the considerations are more or less the same from a VSAN trace perspective. As outlined in the previous post, when an ESXi host booting from one of these devices is also running VSAN, VSAN traces are written to a RAM disk. Since the RAM disk is non-persistent, these logs are written to persistent storage either during host shutdown or on system crash (PANIC). This means that the VSAN traces, which are typically quite write intensive, do not burn out the boot media. This method of first writing the traces to RAM disk and later moving them to persistent store is handled automatically by the ESXi host and there is no user action required. This is the only support method of handling VSAN traces when booting an ESXi from either a USB stick or an SD card. You cannot write VSAN traces directly to SD or USB boot devices at this time.
This brings us onto another flash device known as SATADOMs. SATADOMs, short for Serial ATA Disk on Modules, are basically flash memory modules designed to be inserted into the SATA connector of a server. In VSAN 6.0, ESXi hosts running VSAN are supported when booting from SATADOM, as long as they met specific requirements. On ESXi hosts that boot from SATADOM, the VSAN traces are written directly to the SATADOM. In other words, there is no RAM disk involved. This is why SATADOM specification requirements for SATADOM were documented in the VSAN 6.0 Admin Guide, and the requirement is for an SLC (single level cell) device. The SLCs have higher endurance and quality when compared to other flash devices. the reason for this is once again to prevent any sort of burn-out occurring on the boot device when trace files are being written to it.
Customers who wish to boot their ESXi hosts (participating in VSAN) from a flash device should trade-off the above considerations with VSAN traces on USB, SD versus the cost of a high level SATADOM.
Hope that explains the differences. I have an older post here on SLC/MLC/eMLC if you wish to learn more.