Adventures in VIO, VMware Integrated Openstack

I’ll start this post by stating straight up that I am no OpenStack expert. Far from it. In fact, the only reason I started to play with VMware Integrated OpenStack (VIO) was to get up to speed for a forthcoming OpenStack class that I am taking in next week. What I’ve documented here is a bunch of issues I ran into during the VIO deployment process. Hopefully these will prove useful to some folks who are also new to OpenStack and plan on going through the same exercise. I’m not going to describe VIO in any detail, nor any of the OpenStack components such as Nova, Cinder, Glance, etc. (this is all just a quick google away) but suffice to say that what VMware has done is to bundle OpenStack into a bunch of VMs responsible for running different services, and allows you to deploy and stand-up an OpenStack configuration very quickly indeed.

The process to deploy VIO is two-fold:

1. Deploy the VIO OVA, which deploys a management plugin on the vSphere web client, as well as the OpenStack template. This could be deployed on the same cluster as your VIO management cluster, or it could be deployed to a dedicated management cluster.
2. Provision OpenStack on top of vSphere. This would be deployed on a dedicated cluster of ESXi hosts and becomes your VIO management cluster

After the completion of step 2, you are now ready to provision VMs. The following are some issues that I encountered when rolling out my VIO deployment. They may help you if you have to go through the same exercise.

DRS has not been enabled for vCenter Cluster

This is a pretty simple issue. VIO expects to find that the cluster on which it is deploying has DRS enabled. Enable DRS on the cluster where you plan to deploy VIO to resolve the issue.

Insufficient number of ESXi hosts to meet the anti-affinity rules

For resilience, the services deployed by VIO (such as load balancer and DHCP server) have primary and backup configuration and as such need to be deployed on different hosts in the compute cluster. The database service is deployed as three VMs. You need to make sure that there are enough hosts in the cluster (i.e. at least three,  but I used four hosts) to make sure that these service VMs are not placed on the same host. This means that the services can continue to be available, even when a compute host fails.

Cannot find datastore with enough space and meets the datastore anti-affinity rule

This was an interesting one. I didn’t understand what this was complaining about. I had plenty of hosts, and each host had plenty of local storage. Should that be enough? Apparently not. Upon further investigation, I needed to use shared storage for Nova and Glance datastores. In other words, the datastore needed to be visible to all hosts. Once I created some shared storage and made it accessible to all hosts in the compute cluster, and selected those datastores for the Nova and Glance datastores, the issue was resolved.

Invalid Parameter: DVS, error [IP address] cannot be found in the distributed virtual switch

VIO can use two networking features, either a DVS or NSX. I went with a DVS option rather than NSX. However, I had multiple DVS in this environment, and I chose the incorrect one in this case (I chose the management cluster DVS and not the compute cluster DVS). Going back and selecting the correct DVS resolved the issue.

Network [dvPortgroup] is not in vSphere cluster

This was another fat-finger issue. Instead of choosing portgroups on my VIO management cluster DVS when setting up the management network and the OpenStack API access network, I chose portgroups from my other DVS. Once I correctly chose the correct portgroups when setting up the VIO networks, this issue was resolved. A good tip here would be to label the distributed portgroups so that they are easily identifiable as belonging to the OpenStack deployment.

After addressing these issues, I finally got VIO to deploy:

I was now ready to login to the OpenStack portal and deploy my first VM. However this is where I encountered my next issue.

Error: Failed to create network “net1″:400-{u’NeutronError’:{u’message’: u’Invalid input for operation: Unable to create network.’, u’type’:u’InvalidInput’, u’detail’:u”}}

OK, so this isn’t the most useful error message. What I was trying to do here is launch an instance (i.e. deploy a VM on top of OpenStack). Eventually I found out what this issue was. A Provider Networks backed by VLANs must be created if you are using DVS networking mode. DVS doesn’t have support for any overlay/private tenant networks. Overlay/Private Tenant networks are only available with NSX. Once I created my own provider network, I could successfully launch an instance.

This step is actually in the quick-start guide, but unless you read through to the very end, you won’t see it.

Recap on tips for deploying VIO

• There must be at least 3 ESXi hosts in the VIO cluster
• DRS must be enabled
• Shared storage must be presented to all hosts
• If not using NSX, a DVS must be created for the VIO cluster
• You will need a VM management dvportgroup and an OpenStack API management dvportgroup
• You will need a range of 18 IP address for VM management  and a range of 2 to 5 IP addresses for the Openstack API access network
• The ESXi host firewalls must be configured to allow gdbserver access. Typically, the port range is 5900-5964.
• Make sure to label portgroups on DVS for easy identification
• If using DVS, a Provider Network must be created to allow launching of instances (provisioning of VMs)

So some of these are pretty simple. Others a bit more complex to figure out the root cause. Hopefully this will be useful to someone. Once I take the OpenStack class, I might see if there is more to share with VIO.