ATS Miscompare revisited in vSphere 6.5

Does anyone remember the ATS Miscompare issue? This blog post from 2 years ago might jog your memory. It is basically an issue that arose when we began using ATS, the VAAI Atomic Test and Set primitive, for maintaining the ‘liveness’ of a heartbeat in vSphere 5.5U2.  After making this change, a number of customers started to see “ATS Miscompare detected between test and set HB images” messages after upgrading to vSphere 5.5U2 or later. The HB reference in the message is shorthand for heartbeat. In previous releases, we did not use ATS for maintaining the ‘liveness’ of a heartbeat.…

Improved block allocation mechanism on VMFS-6

Along with other improvements to VMFS-6, there is also a new block allocation mechanism which aims to reduce lock contention between hosts sharing the same VMFS-6 filesystem. To understand how lock contention could arise, it is important to understand that resources on VMFS are grouped into resource clusters; this is the same for VMFS-6 and earlier versions on VMFS. Resources could be file descriptors, sub-block, file block, pointer blocks, and so on. Historically, we have always tried to allocate different resource clusters to different ESXi hosts, which meant that only VMs running on the same host shared resources within the…

A change to sub-blocks on VMFS-6

Something that I only just recently noticed is that we have made a change to the sub-blocks structure on VMFS-6, compared to VMFS-5. Sub-blocks are small allocations on a VMFS volume, and they are used to back small files. They were introduced as a space-saving measure to prevent using a full file block to back a very small file. To put this simply, when a file is created on VMFS, it is initially backed by a sub-block, and when the file grows above the size of a sub-block, it is switched to being backed by a file block (this has…

VMFS-6 Large and Small File Blocks – what are they?

When vSphere 6.5 released towards the end of 2016, it introduced a brand new version of VMFS, VMFS-6. VMFS probably needs little in the way of introduction at this stage, it being VMware’s flagship filesystem for over 10 years at this point. There is an older VMFS whitepaper available here if you are new to VMFS and want to get more of an overview. Now VMFS-6 introduces two new internal block sizes concept for file creation. These are referred to as LFB (Large File Blocks) and SFB (Small File Blocks) and are used to back files on the VMFS-6 volume.…

vSphere 6.5 p01 – Important patch for users of Automated UNMAP

VMware has just announced the release of vSphere 6.5 p01 (Patch ESXi-6.5.0-20170304001-standard). While there are a number of different issues addressed in the patch, there is one in particular that I wanted to bring to your attention. Automated UNMAP is a feature that we introduced in vSphere 6.5. This patch contains a fix for some odd behaviour seen with the new Automated UNMAP feature. The issue has only been observed with certain Guest OS, certain filesystems, and a certain block sizes format. KB article 2148987 for the patch describes it as follows: Tools in guest operating system might send unmap…

Gathering core dump files when encryption is enabled

One of the key new features of vSphere 6.5 is vSphere VM Encryption, a mechanism to encrypt all virtual machine files. This mechanism not only encrypts the VMDK, but also the metadata files and core dumps associated with a VM. Now, there would not be much point in sending an encrypted core dump file to VMware for analysis, so a mechanism has been put in place to allow these files to be recrypted using a password before sending them to VMware. The password can then be shared with VMware to allow us to examine the core dumps.  This is how…

Virtually Speaking Podcast Episode #34 – Core Storage

Many readers will be aware that John Nicholson and Pete Fletcha of the VMware Storage and Availability Tech Marketing team run a weekly podcast show called  Virtually Speaking. This week I am back as a guest on their show, alongside Cody Hosterman of Pure Storage. We discuss a lot of the new core storage features in vSphere 6.5, which were detailed in a co-produced white paper that we recently created. You can read about how to get the white paper here. You can listen to the podcast through the player below. I hope you enjoy it.