Hello from VMworld EMEA in Barcelona. Well, we can finally talk about vSphere 6.5 today. In this post, I want to highlight a number of new and enhanced features that you will find in vSphere 6.5 related to core storage. I am not going to discuss Virtual SAN (VSAN), Virtual Volumes (VVols) or I/O Filter enhancements (VAIO) specifically in this post, although you will no doubt see some new features tie directly into the latter. Instead, I want to talk about those features that are specific to core storage.
A short post to let you know about some upcoming speaking engagements that I am doing over the next couple of weeks.
First up, I will be speaking at the TechUG, or Technology User Group event next week. This event will be held on Thursday, November 26th. It will be held in the Westin Hotel in the heart of Dublin city, Ireland. There is a really good agenda for this event (which is not a VMware centric event), that you can find at this link here. I personally will be speaking about Virtual SAN (VSAN), VMware’s hyper-converged compute and storage platform. This will be more of an introductory type session, but I’ll also be giving an overview of new and upcoming features and where we are thinking about going next with VSAN. You can find the Dublin TechUG registration link here.
My next session is at the VMUGDK Usercon or Nordics Usercon, which will be held on Tuesday, December 1st. This event will take place at the Scandic Hotel in Copenhagen, Denmark. This year I will return to my roots and talk about core vSphere storage enhancements over the past few releases, and also a look at some upcoming plans. No VSAN, VVol or anything like that – this will be a discussion on VMFS, NFS, VAAI, PSA, etc. The Nordic UserCon details can be found at this link here. The registration link is at the same location.
If you are in the Dublin or Copenhagen area for any of these events, I’d love to see you there. I plan to spend most of the day at both events, so if there are any VSAN or vSphere storage questions or feedback that you’d like to give me, I’d be delighted to talk with you in person.
This is a new feature in vSphere 6.0 that I only recently became aware of. Prior to vSphere 6.0, all the I/Os from a given virtual machine to a particular device would share a single I/O queue. This would result in all the I/Os from the VM (boot VMDK, data VMDK, snapshot delta) queued into a single per-VM, per-device queue. This caused I/Os from different VMDKs interfere with each other and could actually hurt fairness.
For example, if a VMDK was used by a database, and this database issued a lot of I/O, this could compete with I/Os from the boot-disk. This in turn could make it appear that the VM (Guest OS) is running slowly.
A very quick “public service announcement” post this morning folks, simply to bring your attention to a new knowledge base article that our support team have published. The issue relates to APD (All Paths Down) which is a condition that can occur when a storage device is removed from an ESXi host in an uncontrolled manner. The issue only affects ESXi 6.0. The bottom line is that even though the paths to the device recover and the device is online, the APD timeout continues to count down and expire, and as a result, the device is placed in APD timeout state. This obviously has an impact on virtual machines, workloads, etc, that are using this device.
Unfortunately there is no resolution at this time, but there are some workarounds detailed in the KB article. For those of you who dealt with APD events in earlier versions of vSphere, you’ll know the drill.
The KB article is 2126021. Note that this doesn’t affect all APD behaviors. Most APD events are handled just fine. However I’d urge you to take a quick read of the KB just to familiarize yourself with the behaviour and workarounds while we work on a permanent solution.
[Update: 10 Sep 2015] This issue is now fixed in ESXi 6.0U1. Details may be found in the ESXi 6.0U1 release notes.
I’ve been hit up this week by a number of folks asking about “ATS Miscompare detected between test and set HB images” messages after upgrading to vSphere 5.5U2 and 6.0. The purpose of this post is to give you some background on why this might have started to happen.
First off, ATS is the Atomic Test and Set primitive which is one of the VAAI primitives. You can read all about VAAI primitives in the white paper. HB is short for heartbeat. This is how ownership of a file (e.g VMDK) is maintained on VMFS, i.e. lock. You can read more about heartbeats and locking in this blog post of mine from a few years back. In a nutshell, the heartbeat region of VMFS is used for on-disk locking, and every host that uses the VMFS volume has its own heartbeat region. This region is updated by the host on every heartbeat. The region that is updated is the time stamp, which tells others that this host is alive. When the host is down, this region is used to communicate lock state to other hosts.
In vSphere 5.5U2, we started using ATS for maintaining the heartbeat. Prior to this release, we only used ATS when the heartbeat state changed. For example, referring to the older blog, we would use ATS in the following cases:
- Acquire a heartbeat
- Clear a heartbeat
- Replay a heartbeat
- Reclaim a heartbeat
We did not use ATS for maintaining the ‘liveness’ of a heartbeat. This is the change that was introduced in 5.5U2 and which appears to have led to issues for certain storage arrays.
Recently I published an article on Virtual Volumes (VVols) where I touched on a comparison between how migrations typically worked with VAAI and how they now work with VVols. In the meantime, I managed to have some really interesting discussions with some of our VVol leads, and I thought it worth sharing here as I haven’t seen this level of detail anywhere else. This is rather a long discussion, as there are a lot of different permutations of migrations that can take place. There are also different states that the virtual machine could be in. We’re solely focused on VVols here, so although different scenarios are offered up, I highlight what scenario we are actually considering.
There was a time when VMFS was the only datastore that could be used with ESXi. That has changed considerably, with the introduction of NFS (v3 and v4.1), Virtual Volumes and of course Virtual SAN. However VMFS continues to be used by a great many VMware customers and of course we look to enhance it with each release of vSphere. This post will cover changes and enhancements to VMFS in vSphere 6.0.