A quick note to let you know about a new KB article that has recently been published which reports incorrect values for Outstanding IO in the VSAN Observer tool used for monitoring performance of VSAN deployments when using vSphere 5.5U2.
Virtual SAN (VSAN) Observer graphs in the “VSAN Client”, “VSAN Disk”, “DOM Owner” or individual VSAN object on the “VM” tab show very high Outstanding I/O (OIO) value that is inconsistent with the actual I/O load.
Here is a sample screenshot from my VSAN environment running vSphere 5.5U2. As you can see the Outstanding IO values are off the scale:
Of course, this behaviour may lead to you “chasing your tail” so to speak when monitoring or troubleshooting VSAN, so we are working on getting this resolved asap. Check the KB article regularly for updates regarding a fix. In the meantime, understand that a high Outstanding IO count in VSAN Observer is expected and may not be the symptom of any underlying issue.
There was a very interesting discussion on our internal forums here at VMware over the past week. One of our guys had built out a VSAN cluster, and everything looked good. However on attempting to deploy a virtual machine on the VSAN datastore, he kept hitting an error which reported that it “cannot complete file creation operation”. As I said, everything looked healthy. The cluster formed correctly, there were no network partitions and the network status was normal. So what could be the problem?
After a great VMworld 2014 in San Francisco, it’s nearly time for the European version which is once again in the beautiful city of Barcelona, Spain. I thought I’d put together a short post on some of my activities, Hopefully you can join me for some.
Introduction to Virtual Volumes – vBrownBag session (Tuesday, October 14th at 2:15pm – Hang Space) I’m delighted to be joined by Nick Dyer of Nimble Storage for an overview of our forthcoming Virtual Volumes. I’ll be sharing our vision for VVols in the whole Software Define Storage story, and Nick will share with you some examples of how Nimble are implementing VVols. Thanks to the vBrownBag team for hosting us.
Maxta are another storage vendor that I managed to get talking to at this years’ VMworld conference in San Francisco. Although they were present at last year’s VMworld, they only announced themselves in earnest last November (11/12/13) with the release of the Maxta Storage Platform (MxSP). I spent some time with Kiran Sreenivasamurthy, Director of PM & PMM at Maxta, and he was very open in sharing details on the Maxta product.
If you read the blurb on Maxta on the VMworld sponsor/exhibitor list, it states that they eliminate the need for storage arrays, provide enterprise class data services and has full virtualization integration from UI to data management.
So on the face of it, Maxta is another converged solution, similar in many respects to VMware’s own Virtual SAN, Nutanix, Simplivity, etc. So what makes Maxta so different? Kiran shared his views with me here.
I have been doing a bunch of stuff around disaster recovery (DR) recently, and my storage of choice at both the production site and the recovery site has been VSAN, VMware Virtual SAN. I have already done a number of tests already with products like vCenter Server, vCenter Operations Manager and NSX, our network virtualization product. Next up was VCO, our vCenter Orchestrator product. I set up vSphere Replication for my vCO servers (I deployed them in a HA configuration) and their associated SQL DB VM on Friday, but when I got in Monday morning, I could not log onto my vCenter. The problem was that my vCenter was running on VSAN (a bit of a chicken and egg type situation), so how do I troubleshoot this situation without my vCenter. And what was the actual problem? Was it a VSAN issue? This is what had to be done to resolve it.
My first introduction to X-IO was via Stephen Foskett’s Tech Field Days. They piqued my interest and I added them to the list of storage vendors that I wanted to check out at VMworld 2014. I started to research these guys a little more, and learnt that they are closely related to Xiotech, a SAN company that I dealt with on occasion when I worked in technical support for VMware back in the day. It seems that Xiotech acquired Seagate’s spun-out Advanced Storage Group in 2007. The guys then began to work on a different product to the Xiotech team, namely the Intelligent Storage Element or ISE array. The Xiotech products were discontinued in 2012 (although the name continues to appear on the VMware SAN/Storage HCL), and the focus was placed on the ISE products. I was a bit confused when I saw that X-IO were not listed on the HCL directly, but after checking with Blair Parkhill, VP of Tech Marketing at X-IO, it seems that they still use their incorporated name, Xiotech.
My colleague Duncan wrote a post relatively recently around the new mclock I/O scheduler which VMware introduced in vSphere 5.5. He also mentioned some caveats with the new scheduler, especially around the I/O size (32K) used with the IOPS setting, which may lead to some unexpected behaviour. As Duncan mentioned, the reason for introducing the new scheduler is primarily to provide a better I/O scheduling mechanism that allows for limits, shares and reservations. Unfortunately, we didn’t do a very good job of announcing this change in I/O scheduling, or documenting the behaviour, and it has led to a number of additional questions from our customers. I hope to address some of these here.