A quick note to let you know about a new KB article that has recently been published which reports incorrect values for Outstanding IO in the VSAN Observer tool used for monitoring performance of VSAN deployments when using vSphere 5.5U2.
Virtual SAN (VSAN) Observer graphs in the “VSAN Client”, “VSAN Disk”, “DOM Owner” or individual VSAN object on the “VM” tab show very high Outstanding I/O (OIO) value that is inconsistent with the actual I/O load.
Here is a sample screenshot from my VSAN environment running vSphere 5.5U2. As you can see the Outstanding IO values are off the scale:
Of course, this behaviour may lead to you “chasing your tail” so to speak when monitoring or troubleshooting VSAN, so we are working on getting this resolved asap. Check the KB article regularly for updates regarding a fix. In the meantime, understand that a high Outstanding IO count in VSAN Observer is expected and may not be the symptom of any underlying issue.
My first introduction to X-IO was via Stephen Foskett’s Tech Field Days. They piqued my interest and I added them to the list of storage vendors that I wanted to check out at VMworld 2014. I started to research these guys a little more, and learnt that they are closely related to Xiotech, a SAN company that I dealt with on occasion when I worked in technical support for VMware back in the day. It seems that Xiotech acquired Seagate’s spun-out Advanced Storage Group in 2007. The guys then began to work on a different product to the Xiotech team, namely the Intelligent Storage Element or ISE array. The Xiotech products were discontinued in 2012 (although the name continues to appear on the VMware SAN/Storage HCL), and the focus was placed on the ISE products. I was a bit confused when I saw that X-IO were not listed on the HCL directly, but after checking with Blair Parkhill, VP of Tech Marketing at X-IO, it seems that they still use their incorporated name, Xiotech.
My colleague Duncan wrote a post relatively recently around the new mclock I/O scheduler which VMware introduced in vSphere 5.5. He also mentioned some caveats with the new scheduler, especially around the I/O size (32K) used with the IOPS setting, which may lead to some unexpected behaviour. As Duncan mentioned, the reason for introducing the new scheduler is primarily to provide a better I/O scheduling mechanism that allows for limits, shares and reservations. Unfortunately, we didn’t do a very good job of announcing this change in I/O scheduling, or documenting the behaviour, and it has led to a number of additional questions from our customers. I hope to address some of these here.
Yesterday was my first day at VMworld 2014. As usual with this event, there are simply so many interesting announcements that it is hard to keep track. However, for me, there were a few things which stood out in the storage space worth calling out. These are specifically VMware focused products and features. I know that many of our partners have also made announcements in the storage space, but for today I concentrated solely on VMware. There are the two that really caught my attention.
Last year I published a list of storage vendors and partners that I was planning to check out at VMworld 2013. This year is no different, with a number of new arrivals on the storage scene, as well as some super new cool products from many of VMware’s partners. Whilst this is no means a definitive list of what’s on show, these are the ones that I am particularly interested in checking out this year.
On a recent trip to VMware in Palo Alto, I found some time to visit with a good pal of mine, Vinay Gaonkar, who is now the Product Manager for XtremIO over at EMC. Vinay used to be a storage PM at VMware (he worked on the initial phases of VVols), and we worked together on a number of storage items in various vSphere releases. It’s been almost 2 years since I last spoke to the XtremIO folks (VMworld 2012 in fact, when the product still had not become generally available), so I thought that this would be a good time to catch up with them, as we are in the run up to VMworld 2014.
I was in a conversation with one of my pals over at Tintri last week (Fintan), and he observed some strange behaviour when provisioning VMs from a catalog in vCloud Director (vCD). When he disabled Fast Provisioning, he expected that provisioning further VMs from the catalog would still be offloaded via the VAAI-NAS plugin. All the ESXi hosts have the VAAI-NAS plugin from Tintri installed. However, it seems that the provisioning/cloning operation was not being offloaded to the array, and the ESXi hosts resources were being used for the operation instead. Deployments of VMs from the catalogs were taking minutes rather than seconds. What was going on?