Kubernetes on vSphere
I’ve talked a lot recently about the various VMware projects surrounding containers, container management, repositories, etc. However one of the most popular container cluster managers is Kubernetes (originally developed by Google). To use an official description, Kubernetes (or K8S for short) is a “platform for automating deployment, scaling, and operations of application containers across clusters of hosts”. So this begs the question about how easy is it to deploy K8S on vSphere. I have already documented how K8S can be deployed on Photon Platform. But can you easily deploy Kubernetes on a vSphere infrastructure. The answer now is that it is relatively easy. This necessary scripts are now included in K8S version 1.4.5, which went live recently (October 29th). Let’s look at the steps involved in deploying Kubernetes on vSphere in more detail.
Step 1. Deploy from where?
The first decision is to figure out where to deploy K8S from. In this example, I am going to roll out a VMware Photon OS VM, and use that as a way to deploy K8S to my vSphere infrastructure. Photon OS can be downloaded as an OVA from here. I used the HW11 version. However you could also deploy this from a laptop or desktop if you so wish.
My infrastructure is 3 hosts running ESXi 6.0u2, managed by a vCenter which is also running 6.0u2. I also have vSAN enabled to provide highly available persistent storage to the hosts.
Step 2. Setting up Photon OS
When you first open an SSH to the Photon OS, you will need to provide the default password of “changeme” and set a new password. There are a number of items that you need to add if you deploy the minimal Photon OS OVA like I have just done.
- Go – Go programming language
- govc – CLI for interacting with VMware vSphere APIs via Go
- awk – parsing utility used by K8S scripts
- tar – needed to extract K8S tar ball
2.1 Go
These are the steps to install Go in your Photon OS VM:
root@photon-qBvwmMUFl [ ~ ]# tdnf install go Installing: mercurial x86_64 3.7.1-3.ph1 31.10 M go x86_64 1.6.3-1.ph1 219.92 M Total installed size: 251.02 M Is this ok [y/N]:y Downloading: go 57584085 100% mercurial 9025599 100% Testing transaction Running transaction Complete! root@photon-qBvwmMUFl [ ~ ]# go version go: cannot find GOROOT directory: /usr/bin/go root@photon-qBvwmMUFl [ ~ ]# mkdir -p $HOME/go root@photon-qBvwmMUFl [ ~ ]# GOROOT=$HOME/go root@photon-qBvwmMUFl [ ~ ]# export GOROOT root@photon-qBvwmMUFl [ ~ ]# go version go version go1.6.3 linux/amd64 root@photon-qBvwmMUFl [ ~ ]#
I would recommend creating a .bash_profile and adding the GOROOT setting. You will need to add other exports shortly, and this will persist them.
2.2 govc
The govc binary can be downloaded from github:
root@photon-qBvwmMUFl [ ~ ]# curl -OL \
https://github.com/vmware/govmomi/releases/download/v0.8.0/govc_linux_amd64.gz
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 595 0 595 0 0 1129 0 --:--:-- --:--:-- --:--:-- 1131
100 7861k 100 7861k 0 0 384k 0 0:00:20 0:00:20 --:--:-- 490k
root@photon-qBvwmMUFl [ ~ ]# gzip -d govc_linux_amd64.gz
root@photon-qBvwmMUFl [ ~ ]# chmod +x govc_linux_amd64
root@photon-qBvwmMUFl [ ~ ]# mv govc_linux_amd64 /usr/local/bin/govc
root@photon-qBvwmMUFl [ ~ ]# govc version
govc 0.8.0
root@photon-qBvwmMUFl [ ~ ]#
2.3 awk and tar
First, we find out which package and repo provides awk, and then install it. Tar can be installed as shown below.
root@photon-qBvwmMUFl [ ~ ]# tdnf whatprovides awk gawk-4.1.3-2.ph1.x86_64 : Contains programs for manipulating text files Repo : photon root@photon-qBvwmMUFl [ ~ ]# tdnf install gawk Installing: mpfr x86_64 3.1.3-2.ph1 501.48 k gawk x86_64 4.1.3-2.ph1 1.89 M Total installed size: 2.38 M Is this ok [y/N]:y Downloading: gawk 790862 100% mpfr 228844 100% Testing transaction Running transaction Complete! root@photon-qBvwmMUFl [ ~ ]# tdnf install tar Installing: tar x86_64 1.28-2.ph1 4.25 M Total installed size: 4.25 M Is this ok [y/N]:y Downloading: tar 1215034 100% Testing transaction Running transaction Complete! root@photon-qBvwmMUFl [ ~ ]#
Step 3. Get Kubernetes and a VMDK image for VMs that will run K8S
There are now two components to pull into our Photon OS that will be required for the K8S deployment. The first is Kubernetes itself, and the second is a VMDK image that will be used to create the Virtual Machines that will run our K8S. These are both going to take a little time due to their sizes.
3.1 Download K8S
root@photon-qBvwmMUFl [ ~ ]# curl -OL \ https://storage.googleapis.com/kubernetes-release/release/v1.4.5/kubernetes.tar.gz % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 1035M 100 1035M 0 0 509k 0 0:34:43 0:34:43 --:--:-- 468k root@photon-qBvwmMUFl [ ~ ]#
3.2 Extract Kubernetes
root@photon-qBvwmMUFl [ ~ ]# tar zxvf kubernetes.tar.gz kubernetes/ kubernetes/server/ kubernetes/server/kubernetes-server-linux-amd64.tar.gz .. . kubernetes/README.md root@photon-qBvwmMUFl [ ~ ]#
3.3 Download the VMDK image
root@photon-qBvwmMUFl [ ~ ]# curl --remote-name-all \ https://storage.googleapis.com/govmomi/vmdk/2016-01-08/kube.vmdk.gz % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 663M 100 663M 0 0 338k 0 0:33:25 0:33:25 --:--:-- 258k root@photon-qBvwmMUFl [ ~ ]#
3.4 Unzip the VMDK image
root@photon-qBvwmMUFl [ ~ ]# gunzip kube.vmdk.gz
Step 4. Setup GO
The next step is to set up a bunch of Go environment variables, so that “govc” runs against your correct environment. Once again, it might be easier to add these to your .bash_profile. I have provided the list of variables specific for my environment here, but you will need to modify them to reflect your setup.
GOVC_URL='10.27.51.103' GOVC_USERNAME='administrator@vsphere.local' GOVC_PASSWORD='*****' GOVC_NETWORK='VM Network' GOVC_INSECURE=1 GOVC_DATASTORE='vsanDatastore' GOVC_RESOURCE_POOL='/CNA-DC/host/Mgmt/Resources' GOVC_GUEST_LOGIN='kube:kube' GOVC_PORT='443' GOVC_DATACENTER='CNA-DC' export GOVC_URL GOVC_USERNAME GOVC_PASSWORD GOVC_NETWORK GOVC_INSECURE \ GOVC_DATASTORE GOVC_RESOURCE_POOL GOVC_GUEST_LOGIN GOVC_PORT GOVC_DATACENTER
There is not too much explaining needed here I think. You will need to provide the correct vCenter password obviously. The resource pool definition is a bit obtuse, but suffice to say that “Mgmt” is the name of my cluster, and the Resource Pool path has to take the format shown here. I am also using vSAN, and so have provided the vsanDatastore as the datastore on which to deploy the VMs that will run K8S. Finally, kube:kube are the credentials associated with the image that we previously downloaded.
Remember to run “source .bash_profile” when you have added these entries.
Step 5. Push the kube.vmdk image to the datastore
Now we use govc to move the kube.vmdk image to the vsanDatastore. We are placing it in the ./kube folder. Afterwards, we list the contents of the folder to make sure it is there.
root@photon-39BgfUQRO [ ~ ]# govc import.vmdk kube.vmdk ./kube/ [09-11-16 16:36:13] Uploading... OK [09-11-16 16:36:49] Importing... OK root@photon-39BgfUQRO [ ~ ]# govc datastore.ls ./kube/ kube.vmdk root@photon-39BgfUQRO [ ~ ]#
You should also be able to navigate to the datastore view in the vSphere UI, and find the VMDK image in the kube folder, as shown here:
Step 6. Create an SSH identity
You need to have an SSH identify to deploy Kubernetes using the “kube-up.sh” method that we are going to use in a moment. These steps show you how to do this.
root@photon-39BgfUQRO [ ~ ]# ssh-keygen -t rsa -b 4096 -C "id" Generating public/private rsa key pair. Enter file in which to save the key (/root/.ssh/id_rsa): Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /root/.ssh/id_rsa. Your public key has been saved in /root/.ssh/id_rsa.pub. The key fingerprint is: SHA256:r7pDWeuz8+wFENryhHtr4/+b/Nx3plHzLaVgScKpDX8 id The key's randomart image is: +---[RSA 4096]----+ | . | | + o . | | + = + . | | =.* o . | | .oS.+ E .o| | o..o + . ++| | . .+ . . + o| | .o++ o o ++| | o++B=.=o+o+| +----[SHA256]-----+ root@photon-39BgfUQRO [ ~ ]# eval $(ssh-agent) Agent pid 666 root@photon-39BgfUQRO [ ~ ]# ssh-add ~/.ssh/id_rsa Identity added: /root/.ssh/id_rsa (/root/.ssh/id_rsa) root@photon-39BgfUQRO [ ~ ]#
Step 7. Roll out Kubernetes using kube-up
We are now ready to deploy K8S. Change directory to the Kubernetes extracted folder, and then run the following command. You need KUBERNETES_PROVIDER set to vsphere, and the kube-up.sh script is in the cluster sub-folder. This is all on the same command line by the way (it is wrapped here just for neatness).
root@photon-39BgfUQRO [ ~/kubernetes ]# KUBERNETES_PROVIDER=vsphere \ cluster/kube-up.sh ... Starting cluster using provider: vsphere ... calling verify-prereqs ... calling kube-up . .
I am not going to reproduce all the output here, but what you should observe is a master VM and 4 minion VMs getting deployed.
You will also see references to K8S being configured via “Salt”, or SaltStack. Salt is a Python-based open-source configuration management software and remote execution engine. Supporting the “Infrastructure as Code” approach to deployment and cloud management, it competes primarily with Puppet, Chef, and Ansible.
If the deployment is successful, you should observer the final output as follows:
. . Found 4 node(s). NAME STATUS AGE kubernetes-minion-1 Ready 3m kubernetes-minion-2 Ready 3m kubernetes-minion-3 Ready 3m kubernetes-minion-4 Ready 3m Validate output: NAME STATUS MESSAGE ERROR scheduler Healthy ok controller-manager Healthy ok etcd-1 Healthy {"health": "true"} etcd-0 Healthy {"health": "true"} Cluster validation succeeded Done, listing cluster services: Kubernetes master is running at https://10.27.51.41 To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
Step 8. Launch the Kubernetes UI
As shown above, you should now be able to connect to the master on whatever IP address is reported in the output. If you append /ui to the URL, and login as “admin”, you should see something like this shown below. To get the admin password, you can find it in the ~/.kube/config file:
root@photon-39BgfUQRO [ ~/kubernetes ]# cat ~/.kube/config | grep password
And there you have it – Kubernetes running on vSphere. In an upcoming post, I’ll include a useful demo which will demonstrate some of K8S features when running on vSphere, especially around persistent and dynamic volumes. But for now, you can hand this off to your developers to get started with K8S.
To shutdown K8S and remove the VMs, simply run “kube-down.sh”:
root@photon-39BgfUQRO [ ~/kubernetes ]# KUBERNETES_PROVIDER=vsphere \
cluster/kube-down.sh
Further reading
There is some additional reading on deploying Kubernetes with vSphere here.
Don’t set goroot, it’s wrong instead set ggopath
I suspect you are correct – I am only doing what Go told me to do.
You don’t need go by the way, you are downloading binary releases not building it
I was under the impression that “govc” needed it. Let me double check.
It seems like I do need “Go”, according to the official K8S docs – http://kubernetes.io/docs/getting-started-guides/vsphere/