vSAN 6.7U1 – Capacity History – Unable to query charts data

A very quick heads-up to let you know about an issue some of us experienced with vSAN 6.7U1 and about how to resolve it. We noticed that after upgrading to vSAN 6.7U1, the new Capacity History view did not work. However brand new installs of 6.7U1 worked just fine. The error that one gets from trying to look at the capacity history is “Unable to query charts data for capacity history” as shown below: I’m pleased to report that we have a solution for this issue already. It was caused by an inability to update the database schema in the…

Heads Up! Misbehaving APD functionality in vSphere 6.0

A very quick “public service announcement” post this morning folks, simply to bring your attention to a new knowledge base article that our support team have published. The issue relates to APD (All Paths Down) which is a condition that can occur when a storage device is removed from an ESXi host in an uncontrolled manner. The issue only affects ESXi 6.0. The bottom line is that even though the paths to the device recover and the device is online, the APD timeout continues to count down and expire, and as a result, the device is placed in APD timeout…

Heads Up! ATS Miscompare detected between test and set HB images

I’ve been hit up this week by a number of folks asking about “ATS Miscompare detected between test and set HB images” messages after upgrading to vSphere 5.5U2 and 6.0. The purpose of this post is to give you some background on why this might have started to happen. First off, ATS is the Atomic Test and Set primitive which is one of the VAAI primitives. You can read all about VAAI primitives in the white paper. HB is short for heartbeat. This is how ownership of a file (e.g VMDK) is maintained on VMFS, i.e. lock. You can read…

ESXi 5.5 EP6 is now live. Important patch for VSAN users

I wouldn’t normally call out new patch releases in my blog, but this one has an important fix for Virtual SAN users. As per KB article 2102046, this patch addresses a known issue with clomd. The symptoms are as follows: Virtual machine operations on the Virtual SAN datastore might fail with an error message similar to the following: create directory <server-detail>-<vm-name> (Cannot Create File) The clomd service might also stop responding. Virtual SAN cluster might report that the Virtual SAN datastore is running out of space even though space is available in the datastore. An error message similar to the…

Heads Up! Incorrect reporting of Outstanding IO in VSAN Observer

Hi all, A quick note to let you know about a new KB article that has recently been published which reports incorrect values for Outstanding IO in the VSAN Observer tool used for monitoring performance of VSAN deployments when using vSphere 5.5U2. KB 2091979 reports the issue as follows: Virtual SAN (VSAN) Observer graphs in the “VSAN Client”, “VSAN Disk”, “DOM Owner” or individual VSAN object on the “VM” tab show very high Outstanding I/O (OIO) value that is inconsistent with the actual I/O load. Here is a sample screenshot from my VSAN environment running vSphere 5.5U2. As you can see…

Heads Up! VASA Storage Providers disconnected – VSAN Capabilities Missing

I’m a bit late in bringing this to your attention, but there is a potential issue with VASA storage providers disconnecting from vCenter resulting in no VSAN capabilities being visible when you try to create a VM Storage Policy. These storage providers (there is one on each ESXi host participating in the VSAN Cluster) provide out-of-band information about the underlying storage system, in this case VSAN. If there isn’t at least one of these providers on the ESXi hosts communicating to the SMS (Storage Monitoring Service) on vCenter, then vCenter will not be able to display any of the capabilities…

Heads Up! HP Smart Array Driver Issue

We’ve seen a spate of incidents recently related to the HP Smart Array Drivers that are shipped as part of ESXi 5.x. Worst case scenario – this is leading to out of memory conditions and a PSOD (Purple Screen of Death) on the ESXi host in some cases. The bug is in the hpsa 5.5.0.58-1 driver and all Smart Array controllers that use this driver are exposed to this issue. For details on the symptom, check out VMware KB article 2075978. HP have also released a Customer Advisory c04302261 on the issue. This was a tricky one to deal with, as…