A very quick “public service announcement” post this morning folks, simply to bring your attention to a new knowledge base article that our support team have published. The issue relates to APD (All Paths Down) which is a condition that can occur when a storage device is removed from an ESXi host in an uncontrolled manner. The issue only affects ESXi 6.0. The bottom line is that even though the paths to the device recover and the device is online, the APD timeout continues to count down and expire, and as a result, the device is placed in APD timeout state. This obviously has an impact on virtual machines, workloads, etc, that are using this device.
Unfortunately there is no resolution at this time, but there are some workarounds detailed in the KB article. For those of you who dealt with APD events in earlier versions of vSphere, you’ll know the drill.
The KB article is 2126021. Note that this doesn’t affect all APD behaviors. Most APD events are handled just fine. However I’d urge you to take a quick read of the KB just to familiarize yourself with the behaviour and workarounds while we work on a permanent solution.
I’ve been hit up this week by a number of folks asking about “ATS Miscompare detected between test and set HB images” messages after upgrading to vSphere 5.5U2 and 6.0. The purpose of this post is to give you some background on why this might have started to happen.
In vSphere 5.5U2, we started using ATS for maintaining the heartbeat. Prior to this release, we only used ATS when the heartbeat state changed. For example, referring to the older blog, we would use ATS in the following cases:
Acquire a heartbeat
Clear a heartbeat
Replay a heartbeat
Reclaim a heartbeat
We did not use ATS for maintaining the ‘liveness’ of a heartbeat. This is the change that was introduced in 5.5U2 and which appears to have led to issues for certain storage arrays.
Recently I published an article on Virtual Volumes (VVols) where I touched on a comparison between how migrations typically worked with VAAI and how they now work with VVols. In the meantime, I managed to have some really interesting discussions with some of our VVol leads, and I thought it worth sharing here as I haven’t seen this level of detail anywhere else. This is rather a long discussion, as there are a lot of different permutations of migrations that can take place. There are also different states that the virtual machine could be in. We’re solely focused on VVols here, so although different scenarios are offered up, I highlight what scenario we are actually considering.
There was a time when VMFS was the only datastore that could be used with ESXi. That has changed considerably, with the introduction of NFS (v3 and v4.1), Virtual Volumes and of course Virtual SAN. However VMFS continues to be used by a great many VMware customers and of course we look to enhance it with each release of vSphere. This post will cover changes and enhancements to VMFS in vSphere 6.0.
I was involved in some conversations recently on how the VAAI UNMAP command behaved, and what were the characteristics which affected its performance. For those of you who do not know, UNMAP is our mechanism for reclaiming dead or stranded space from thinly provisioned VMFS volumes. Prior to this capability, the ESXi host had no way of informing the storage array that the space that was being previously consumed by a particular VM or file is no longer in use. This meant that the array thought that more space was being consumed than was actually the case. UNMAP, part of the vSphere APIs for Array Integration, enables administrators to overcome tho challenge by telling the array that these blocks on a thin provisioned volume are no longer in use and that they can be reclaimed.
There are many occasions where the information displayed in the vSphere client is not sufficient to display all relevant information about a particular storage device, or indeed to troubleshoot problems related to a storage device. The purpose of this post is to explain some of the most often used ESXCLI commands that I use when trying to determine storage device information, and to troubleshoot a particular device.
Continuing on the series of vSphere 5.5 Storage Enhancements, we now come to a feature that is close to many people’s hearts. The vSphere Storage API for Array Integration (VAAI) UNMAP primitive reclaims dead or stranded space on a thinly provisioned VMFS volume, something that we could not do before this primitive came into existence. However, it has a long and somewhat checkered history. Let me share the timeline with you before I get into what improvements we made in vSphere 5.5.