Heads Up! Incorrect reporting of Outstanding IO in VSAN Observer

Hi all,

A quick note to let you know about a new KB article that has recently been published which reports incorrect values for Outstanding IO in the VSAN Observer tool used for monitoring performance of VSAN deployments when using vSphere 5.5U2.

KB 2091979 reports the issue as follows:

Virtual SAN (VSAN) Observer graphs in the “VSAN Client”, “VSAN Disk”, “DOM Owner” or individual VSAN object on the “VM” tab show very high Outstanding I/O (OIO) value that is inconsistent with the actual I/O load.

Here is a sample screenshot from my VSAN environment running vSphere 5.5U2. As you can see the Outstanding IO values are off the scale:

OutstandingIOOf course, this behaviour may lead to you “chasing your tail” so to speak when monitoring or troubleshooting VSAN, so we are working on getting this resolved asap. Check the KB article regularly for updates regarding a fix. In the meantime, understand that a high Outstanding IO count in VSAN Observer is expected and may not be the symptom of any underlying issue.

VSAN Part 29 – Cannot complete file creation operation

essential-vsan-bookThere was a very interesting discussion on our internal forums here at VMware over the past week. One of our guys had built out a VSAN cluster, and everything looked good. However on attempting to deploy a virtual machine on the VSAN datastore, he kept hitting an error which reported that it “cannot complete file creation operation”. As I said, everything looked healthy. The cluster formed correctly, there were no network partitions and the network status was normal. So what could be the problem?

Continue reading

My VMworld 2014 – Barcelona (#STO2996-SPO #TEX1985)

vmworld-2014-active-banner-v4After a great VMworld 2014 in San Francisco, it’s nearly time for the European version which is once again in the beautiful city of Barcelona, Spain.  I thought I’d put together a short post on some of my activities, Hopefully you can join me for some.

Introduction to Virtual Volumes –  vBrownBag session (Tuesday, October 14th at 2:15pm – Hang Space) I’m delighted to be joined by Nick Dyer of Nimble Storage for an overview of our forthcoming Virtual Volumes. I’ll be sharing our vision for VVols in the whole Software Define Storage story, and Nick will share with you some examples of how Nimble are implementing VVols. Thanks to the vBrownBag team for hosting us.

Continue reading

DR of VMware vCenter Orchestrator

VCOOver the past month or so, I’ve been looking at disaster recovery of some of the vCloud Suite components. My experiences of using vSphere Replication and Site Recovery Manager to protect and recover vCenter Operations Manager in the event of a disaster can be found here and here. Now it was time to look at vCenter Orchestrator (vCO) to see if that could be protected and recovered.

In this configuration, I deployed vCO in HA mode, meaning that there were two vCenter Orchestrator servers, one running and one in standby mode. The database for vCO was an external SQL Server database, running in its own VM. So there were three VMs to protect in this setup.

Continue reading

VSAN Troubleshooting Case Study

vsan-vmware-virtual-san-boxI have been doing a bunch of stuff around disaster recovery (DR) recently, and my storage of choice at both the production site and the recovery site has been VSAN, VMware Virtual SAN. I have already done a number of tests already with products like vCenter Server, vCenter Operations Manager and NSX, our network virtualization product. Next up was VCO, our vCenter Orchestrator product. I set up vSphere Replication for my vCO servers (I deployed them in a HA configuration) and their associated SQL DB VM on Friday, but when I got in Monday morning, I could not log onto my vCenter. The problem was that my vCenter was running on VSAN (a bit of a chicken and egg type situation), so how do I troubleshoot this situation without my vCenter. And what was the actual problem? Was it a VSAN issue? This is what had to be done to resolve it.

Continue reading

Some useful NSX Troubleshooting Tips

nsx-logoI’ve been working on some Disaster Recovery (DR) scenarios recently with my good pal Paudie. Last month we looked at how we might be able to protect vCenter Operation Manager, by using a vApp construct and also using IP customization. After VMworld, we turned our attention to NSX, and how we might be able to implement a DR solution for NSX. This is still a work in progress, but we did learn some very useful NSX troubleshooting commands that I thought would be worth sharing with you.

Continue reading

PowerCLI 5.8 Release 1 – new SPBM cmdlets

powercli58Yesterday saw the release of vCloud Suite 5.8. While there are quite a few new enhancements to the VMware product line in this release, what really jumped out at me were the new PowerCLI cmdlets for Storage Policy Based Management (SPBM). SPBM is a critical component of VIrtual SAN (VSAN), and will play a major role in the Virtual Volumes (VVols) feature which has been tech previewed at VMworld 2014. VVols will enable our storage array partners to implement out Software Defined Storage vision –  you can read more about there. So what are the new cmdlets for PowerCLI in the 5.8 release?

Continue reading