Adventures in VIO, VMware Integrated Openstack

openstackI’ll start this post by stating straight up that I am no OpenStack expert. Far from it. In fact, the only reason I started to play with VMware Integrated OpenStack (VIO) was to get up to speed for a forthcoming OpenStack class that I am taking in next week. What I’ve documented here is a bunch of issues I ran into during the VIO deployment process. Hopefully these will prove useful to some folks who are also new to OpenStack and plan on going through the same exercise. I’m not going to describe VIO in any detail, nor any of the OpenStack components such as Nova, Cinder, Glance, etc. (this is all just a quick google away) but suffice to say that what VMware has done is to bundle OpenStack into a bunch of VMs responsible for running different services, and allows you to deploy and stand-up an OpenStack configuration very quickly indeed.

The process to deploy VIO is two-fold:

  1. Deploy the VIO OVA, which deploys a management plugin on the vSphere web client, as well as the OpenStack template. This could be deployed on the same cluster as your VIO management cluster, or it could be deployed to a dedicated management cluster.
  2. Provision OpenStack on top of vSphere. This would be deployed on a dedicated cluster of ESXi hosts and becomes your VIO management cluster

After the completion of step 2, you are now ready to provision VMs. The following are some issues that I encountered when rolling out my VIO deployment. They may help you if you have to go through the same exercise.

DRS has not been enabled for vCenter Cluster

vio-error1-no-drsThis is a pretty simple issue. VIO expects to find that the cluster on which it is deploying has DRS enabled. Enable DRS on the cluster where you plan to deploy VIO to resolve the issue.

Insufficient number of ESXi hosts to meet the anti-affinity rules

vio-error3-insufficient-hostsFor resilience, the services deployed by VIO (such as load balancer and DHCP server) have primary and backup configuration and as such need to be deployed on different hosts in the compute cluster. The database service is deployed as three VMs. You need to make sure that there are enough hosts in the cluster (i.e. at least three,  but I used four hosts) to make sure that these service VMs are not placed on the same host. This means that the services can continue to be available, even when a compute host fails.

Cannot find datastore with enough space and meets the datastore anti-affinity rule

vio-error4-datastore not mapped to all hostsThis was an interesting one. I didn’t understand what this was complaining about. I had plenty of hosts, and each host had plenty of local storage. Should that be enough? Apparently not. Upon further investigation, I needed to use shared storage for Nova and Glance datastores. In other words, the datastore needed to be visible to all hosts. Once I created some shared storage and made it accessible to all hosts in the compute cluster, and selected those datastores for the Nova and Glance datastores, the issue was resolved.

Invalid Parameter: DVS, error [IP address] cannot be found in the distributed virtual switch

vio-error2-wrong-dvsVIO can use two networking features, either a DVS or NSX. I went with a DVS option rather than NSX. However, I had multiple DVS in this environment, and I chose the incorrect one in this case (I chose the management cluster DVS and not the compute cluster DVS). Going back and selecting the correct DVS resolved the issue.

Network [dvPortgroup] is not in vSphere cluster

vio-error5-network-errorThis was another fat-finger issue. Instead of choosing portgroups on my VIO management cluster DVS when setting up the management network and the OpenStack API access network, I chose portgroups from my other DVS. Once I correctly chose the correct portgroups when setting up the VIO networks, this issue was resolved. A good tip here would be to label the distributed portgroups so that they are easily identifiable as belonging to the OpenStack deployment.

After addressing these issues, I finally got VIO to deploy:

rollout-openstack1I was now ready to login to the OpenStack portal and deploy my first VM. However this is where I encountered my next issue.

Error: Failed to create network “net1″:400-{u’NeutronError’:{u’message’: u’Invalid input for operation: Unable to create network.’, u’type’:u’InvalidInput’, u’detail’:u”}}

OK, so this isn’t the most useful error message. What I was trying to do here is launch an instance (i.e. deploy a VM on top of OpenStack). Eventually I found out what this issue was. A Provider Networks backed by VLANs must be created if you are using DVS networking mode. DVS doesn’t have support for any overlay/private tenant networks. Overlay/Private Tenant networks are only available with NSX. Once I created my own provider network, I could successfully launch an instance.

openstack3This step is actually in the quick-start guide, but unless you read through to the very end, you won’t see it.

Recap on tips for deploying VIO

  • There must be at least 3 ESXi hosts in the VIO cluster
  • DRS must be enabled
  • Shared storage must be presented to all hosts
  • If not using NSX, a DVS must be created for the VIO cluster
  • You will need a VM management dvportgroup and an OpenStack API management dvportgroup
  • You will need a range of 18 IP address for VM management  and a range of 2 to 5 IP addresses for the Openstack API access network
  • The ESXi host firewalls must be configured to allow gdbserver access. Typically, the port range is 5900-5964.
  • Make sure to label portgroups on DVS for easy identification
  • If using DVS, a Provider Network must be created to allow launching of instances (provisioning of VMs)

So some of these are pretty simple. Others a bit more complex to figure out the root cause. Hopefully this will be useful to someone. Once I take the OpenStack class, I might see if there is more to share with VIO.

14 Replies to “Adventures in VIO, VMware Integrated Openstack”

  1. Excellent post Cormac – great reference, you have captured the most frequently hit gotchas when deploying VMware Integrated OpenStack.

    1. Thanks for this post Cormac. There are a number of caveats when deploying with the DVS and not NSX, and I think that you are starting to uncover some of them (but I fear not all).

      This provider network you created – was that done through OpenStack? What how does this correspond to the portgroups on the ESXi hosts?

      Did you instance receive an IP address? And what about DNS resolution?

      1. Hi Maish,

        Thanks for commenting.

        The provider was created using the steps at the very end of the VIO Quickstart guide.

        The instances did indeed received DHCP Address from my 192.168.90.x range that I had assigned.

        I’m afraid I didn’t bother to much with DNS as I just wanted to get some simple hands on.

        Hope this helps.

        1. Thanks Cormac for the speedy reply. and this DHCP was a part of Neutron – or a DHCP server that was already serving this portgroup or is it part of VIO?

  2. Thanks Cormac! We are reviewing the VIO error messages to try to make the unclear ones more clear. Always happy to get feedback – especially from folks who are *not* openstack experts to make this as easy as possible

  3. You said that you are going for openstack Training.
    Is this a VMware Openstack Training and will VMware release the same for General Public.

  4. Thanks Cormac. In addition to minimum three hosts requirement for Anti affinity error, all three hosts must have at least 8 CPU as the DB VMs have 8CPU.

    I had a cluster with 3 hosts out of which one ESX had 2 CPU [ it was a nested VM 🙂 ] and in spite of having 3 hosts in cluster i was getting “Insufficient number of ESXi hosts to meet the anti-affinity rules” error.

    After bumping up the CPU of third ESX to 8 the error was gone.

Comments are closed.