There was a very interesting discussion on our internal forums here at VMware over the past week. One of our guys had built out a VSAN cluster, and everything looked good. However on attempting to deploy a virtual machine on the VSAN datastore, he kept hitting an error which reported that it “cannot complete file creation operation”. As I said, everything looked healthy. The cluster formed correctly, there were no network partitions and the network status was normal. So what could be the problem?
Over the past month or so, I’ve been looking at disaster recovery of some of the vCloud Suite components. My experiences of using vSphere Replication and Site Recovery Manager to protect and recover vCenter Operations Manager in the event of a disaster can be found here and here. Now it was time to look at vCenter Orchestrator (vCO) to see if that could be protected and recovered.
In this configuration, I deployed vCO in HA mode, meaning that there were two vCenter Orchestrator servers, one running and one in standby mode. The database for vCO was an external SQL Server database, running in its own VM. So there were three VMs to protect in this setup.
I have been doing a bunch of stuff around disaster recovery (DR) recently, and my storage of choice at both the production site and the recovery site has been VSAN, VMware Virtual SAN. I have already done a number of tests already with products like vCenter Server, vCenter Operations Manager and NSX, our network virtualization product. Next up was VCO, our vCenter Orchestrator product. I set up vSphere Replication for my vCO servers (I deployed them in a HA configuration) and their associated SQL DB VM on Friday, but when I got in Monday morning, I could not log onto my vCenter. The problem was that my vCenter was running on VSAN (a bit of a chicken and egg type situation), so how do I troubleshoot this situation without my vCenter. And what was the actual problem? Was it a VSAN issue? This is what had to be done to resolve it.
I’ve been working on some Disaster Recovery (DR) scenarios recently with my good pal Paudie. Last month we looked at how we might be able to protect vCenter Operation Manager, by using a vApp construct and also using IP customization. After VMworld, we turned our attention to NSX, and how we might be able to implement a DR solution for NSX. This is still a work in progress, but we did learn some very useful NSX troubleshooting commands that I thought would be worth sharing with you.
A short note to clarify something that has come up a number of times in recent weeks here at VMware. There have been a number of discussions about whether or not we support NFS over IPv6 on vSphere 5.x, and again, on whether or not we support the VAAI-NAS primitives in the same context.
VAAI is an API for offloading tasks to the storage array, but for offloading tasks to NAS arrays, storage vendors need to create their own plugins for the ESXi hosts to achieve this. You can learn more about VAAI-NAS by clicking here. So what about IPv6 support and NFS? And VAAI-NAS?
I guess the next big tech preview at this year’s VMworld was around Virtual Volumes. Yes, we’ve done this before, but this year there were so many vendors showing demos of their VVol implementation, and so many presentations/sessions on the topic that I believe folks are beginning to realize that we are very close indeed to finally having this feature ready. It’s hard to believe that this was first discussed at VMworld 2011, and I alluded to this when I presented a VVol session that I co-delivered with the folks from Nimble Storage at this year’s VMworld.
This topic is going to be huge, and it is going to change the way storage is designed for virtualization environments. And I think almost every storage partner I spoke to at the show is on-board. In fact, “The Register” asked the question last month if there was ANY storage vendor that wasn’t doing a VVol implementation?
I’m not going to get into any technical details of Virtual Volumes in this post – there will be lots of time for this later. Instead, I’m going to share how we believe VVols will now enable Software Defined Storage (SDS) for SAN & NAS arrays, similar to how we have achieved this already with VSAN.
Earlier this week I spoke about our efforts to failover vCenter Operations Manager (vCops) between two sites. In that article I stated that we used vApp containers at DR site, and added vApp variables to the Analytics and UI VMs at the recovery site. While this was painstaking to set up initially, it did provide us with the ability to failover vCops seamlessly to the DR site, with the vApp VMs inheriting their network settings via the vApp construct. At the end of that post, I mentioned a KB article, 2031891, which discusses the DR of vCops using IP Customization via SRM Recovery Plans. This also used a Resource Pool to hold the vCops VMs rather than a vApp. I will cover our experiences with that approach in this post.