I watched a very cool demonstration this morning from the All Flash Array vendor, SolidFire. I spoke with SolidFire at the end of last year, and did a blog post about them here. One of the most interesting parts of our conversation last year was how SolidFire’s QoS feature and VMware’s Storage I/O Control (SIOC) feature could inter-operate. In a nutshell, QoS work at the datastore/volume layer whereas SIOC deals with the VM/VMDK layer. Last week, Aaron Delp and Adam Carter of SolidFire did an introduction to QoS, both on vSphere and on the SolidFire system. And they also did one of the coolest demos that I’d seen in some time, namely how they have managed to get SIOC and QoS to work in tandem.
We just got notification about a potential issue with the VAAI UNMAP primitive when used on EMC VMAX storage systems with Enginuity version 5876.159.102 or later. It seems that during an ESXi reboot, or during a device ATTACH operation, the ESXi may report corruption. The following is an overview of the details found in EMC KB 184320. Other symptoms include vCenter operations on virtual machines fail to complete and the following errors might be found in the VMkernel logs:
WARNING: Res3: 6131: Invalid clusterNum: expected 2624, read 0 [type 1] Invalid totalResources 0 (cluster 0).[type 1] Invalid nextFreeIdx 0 (cluster 0).
WARNING: Res3: 3155: Volume aaaaaaaa-bbbbbbbb-cccc-dddddddddddd ("datastore1") might be damaged on the disk. Resource cluster metadata corruption has been detected
My good pal Duco Jaspars pinged me earlier this week about an issue that was getting a lot of discussion in the VMware community. Duco also pointed me to a blog post by Andreas Peetz where he described the issue in detail here.
The symptom is that the ESXi hostd process becomes unresponsive when software iSCSI is enabled. There is another symptom where an ESXi boot hangs after message “iscsi_vmk loaded successfully” or “vmkibft loaded successfully”. This has only been only observed with the ESXi 5.5 U1 Driver Rollup ISO. It has not been reported by customers using the standard ESXi 5.5U1 media. The VMware ESXi 5.5 Update 1 Driver Rollup provides an installable ESXi ISO image that includes drivers for various products produced by VMware partners.
Initially it was reported in the community that it appeared to be an issue with the Diablo TeraDimm driver that was shipped as part of the roll-up. However further investigation has concluded that the Emulex
be2iscsi driver is at fault and is the root cause. VMware support are recommending that you use an updated be2iscsi driver as per KB article 2075171 to address the issue.
I thought it might be useful to share some of the various VM Storage Policy status that I have observed whilst testing Virtual SAN (VSAN). I’m sure this is by no means a complete list but as I said, these are the ones that I have come across and I am sure these are the status that you will observe most often too.
Continuing on my set of posts related to Virtual SAN (VSAN) interoperability, let’s take a look at how vCenter Operations Manager (vC Ops for short) integrates with Virtual SAN. vC Ops version 5.8, which was released in December 2013, recognizes the VSAN datastore and can report various characteristics, as you might expect. Although vC Ops 5.8 was released around 3 months before VSAN GA’ed, this release works with ESXi 5.5U1 and vCenter 5.5U1, the vSphere release which introduced VSAN. However, this release of vC Ops does not present all the ‘storage’ metrics for VSAN like it does for datastores based on other storage types. But, having said that, there are still a number of useful vC Ops views and metrics that you might find useful which this post will cover.
At this stage, VSAN has only been in GA for a number of weeks, even though many of us here at VMware have been working on it for a year or two (or even more). Sometimes when we get into explaining the details of storage objects, components, etc, we forget that this is all so new for so many people. In a recent post, someone asked me to explain the concept of a witness on VSAN. Looking back over my posts, I was surprised to realize that I hadn’t already explained it. That is the purpose of this post – explain what a witness disk is in VSAN, and what role it provides.
In a previous post I spoke in-depth about the different objects which go to make up a virtual machine which resides on a VSAN datastore. To recap, these are the VM Home Namespace, the VM Swap, the VMDK objects and the snapshot delta objects. Now, VMDKs comply with the full set of rules that are placed in a VM Storage Policy and applied to a virtual machine. Snapshot deltas inherit the same VM Storage Policies as their VMDK base disk and also comply with the full set of rules in the VM Storage Policy – so far so good. VM Home Namespace is a little different – its behaviour and which capabilities it complies with are discussed in this earlier article. This leaves the VM Swap object, and that is what I plan to cover in this article.