Kubernetes on vSphere

kubernetesI’ve talked a lot recently about the various VMware projects surrounding containers, container management, repositories, etc. However one of the most popular container cluster managers is Kubernetes (originally developed by Google). To use an official description, Kubernetes (or K8S for short) is a “platform for automating deployment, scaling, and operations of application containers across clusters of hosts”. So this begs the question about how easy is it to deploy K8S on vSphere. I have already documented how K8S can be deployed on Photon Platform. But can you easily deploy Kubernetes on a vSphere infrastructure. The answer now is that it is relatively easy. This necessary scripts are now included in K8S version 1.4.5, which went live recently (October 29th). Let’s look at the steps involved in deploying Kubernetes on vSphere in more detail.

 Step 1. Deploy from where?

The first decision is to figure out where to deploy K8S from. In this example, I am going to roll out a VMware Photon OS VM, and use that as a way to deploy K8S to my vSphere infrastructure. Photon OS can be downloaded as an OVA from here. I used the HW11 version. However you could also deploy this from a laptop or desktop if you so wish.

My infrastructure is 3 hosts running ESXi 6.0u2, managed by a vCenter which is also running 6.0u2. I also have vSAN enabled to provide highly available persistent storage to the hosts.

Step 2. Setting up Photon OS

When you first open an SSH to the Photon OS, you will need to provide the default password of “changeme” and set a new password. There are a number of items that you need to add if you deploy the minimal Photon OS OVA like I have just done.

  • Go – Go programming language
  • govc – CLI for interacting with VMware vSphere APIs via Go
  • awk – parsing utility used by K8S scripts
  • tar – needed to extract K8S tar ball

2.1 Go

These are the steps to install Go in your Photon OS VM:

root@photon-qBvwmMUFl [ ~ ]# tdnf install go

mercurial x86_64 3.7.1-3.ph1 31.10 M
go x86_64 1.6.3-1.ph1 219.92 M

Total installed size: 251.02 M
Is this ok [y/N]:y

go 57584085 100%
mercurial 9025599 100%
Testing transaction
Running transaction

root@photon-qBvwmMUFl [ ~ ]# go version
go: cannot find GOROOT directory: /usr/bin/go
root@photon-qBvwmMUFl [ ~ ]# mkdir -p $HOME/go
root@photon-qBvwmMUFl [ ~ ]# GOROOT=$HOME/go
root@photon-qBvwmMUFl [ ~ ]# export GOROOT
root@photon-qBvwmMUFl [ ~ ]# go version
go version go1.6.3 linux/amd64
root@photon-qBvwmMUFl [ ~ ]#

I would recommend creating a .bash_profile and adding the GOROOT setting. You will need to add other exports shortly, and this will persist them.

2.2 govc

The govc binary can be downloaded from github:

root@photon-qBvwmMUFl [ ~ ]# curl -OL \
 % Total % Received % Xferd Average Speed Time Time Time Current
 Dload Upload Total Spent Left Speed
100 595 0 595 0 0 1129 0 --:--:-- --:--:-- --:--:-- 1131
100 7861k 100 7861k 0 0 384k 0 0:00:20 0:00:20 --:--:-- 490k
root@photon-qBvwmMUFl [ ~ ]# gzip -d govc_linux_amd64.gz
root@photon-qBvwmMUFl [ ~ ]# chmod +x govc_linux_amd64
root@photon-qBvwmMUFl [ ~ ]# mv govc_linux_amd64 /usr/local/bin/govc
root@photon-qBvwmMUFl [ ~ ]# govc version
govc 0.8.0
root@photon-qBvwmMUFl [ ~ ]#

2.3 awk and tar

First, we find out which package and repo provides awk, and then install it. Tar can be installed as shown below.

root@photon-qBvwmMUFl [ ~ ]# tdnf whatprovides awk
gawk-4.1.3-2.ph1.x86_64 : Contains programs for manipulating text files
Repo     : photon
root@photon-qBvwmMUFl [ ~ ]# tdnf install gawk

mpfr                   x86_64            3.1.3-2.ph1                501.48 k
gawk                   x86_64            4.1.3-2.ph1                  1.89 M

Total installed size: 2.38 M
Is this ok [y/N]:y

gawk                                    790862    100%
mpfr                                    228844    100%
Testing transaction
Running transaction

root@photon-qBvwmMUFl [ ~ ]# tdnf install tar

tar                    x86_64            1.28-2.ph1                  4.25 M

Total installed size: 4.25 M
Is this ok [y/N]:y

tar                                    1215034    100%
Testing transaction
Running transaction

root@photon-qBvwmMUFl [ ~ ]# 

Step 3. Get Kubernetes and a VMDK image for VMs that will run K8S

There are now two components to pull into our Photon OS that will be required for the K8S deployment. The first is Kubernetes itself, and the second is a VMDK image that will be used to create the Virtual Machines that will run our K8S. These are both going to take a little time due to their sizes.

3.1 Download K8S

root@photon-qBvwmMUFl [ ~ ]# curl -OL \
 % Total % Received % Xferd Average Speed Time Time Time Current
 Dload Upload Total Spent Left Speed
100 1035M 100 1035M 0 0 509k 0 0:34:43 0:34:43 --:--:-- 468k
root@photon-qBvwmMUFl [ ~ ]#

3.2 Extract Kubernetes

root@photon-qBvwmMUFl [ ~ ]# tar zxvf kubernetes.tar.gz
root@photon-qBvwmMUFl [ ~ ]#

3.3 Download the VMDK image

root@photon-qBvwmMUFl [ ~ ]# curl --remote-name-all \
 % Total % Received % Xferd Average Speed Time Time Time Current
 Dload Upload Total Spent Left Speed
100 663M 100 663M 0 0 338k 0 0:33:25 0:33:25 --:--:-- 258k
root@photon-qBvwmMUFl [ ~ ]#

3.4 Unzip the VMDK image

root@photon-qBvwmMUFl [ ~ ]# gunzip kube.vmdk.gz

Step 4. Setup GO

The next step is to set up a bunch of Go environment variables, so that “govc” runs against your correct environment. Once again, it might be easier to add these to your .bash_profile. I have provided the list of variables specific for my environment here, but you will need to modify them to reflect your setup.



There is not too much explaining needed here I think. You will need to provide the correct vCenter password obviously. The resource pool definition is a bit obtuse, but suffice to say that “Mgmt” is the name of my cluster, and the Resource Pool path has to take the format shown here. I am also using vSAN, and so have provided the vsanDatastore as the datastore on which to deploy the VMs that will run K8S. Finally, kube:kube are the credentials associated with the image that we previously downloaded.

Remember to run “source .bash_profile” when you have added these entries.

Step 5. Push the kube.vmdk image to the datastore

Now we use govc to move the kube.vmdk image to the vsanDatastore. We are placing it in the ./kube folder. Afterwards, we list the contents of the folder to make sure it is there.

root@photon-39BgfUQRO [ ~ ]# govc import.vmdk kube.vmdk ./kube/
[09-11-16 16:36:13] Uploading... OK
[09-11-16 16:36:49] Importing... OK
root@photon-39BgfUQRO [ ~ ]# govc datastore.ls ./kube/
root@photon-39BgfUQRO [ ~ ]#

You should also be able to navigate to the datastore view in the vSphere UI, and find the VMDK image in the kube folder, as shown here:

k8s-on-vsphere-1Step 6. Create an SSH identity

You need to have an SSH identify to deploy Kubernetes using the “kube-up.sh” method that we are going to use in a moment. These steps show you how to do this.

root@photon-39BgfUQRO [ ~ ]# ssh-keygen -t rsa -b 4096 -C "id"
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:r7pDWeuz8+wFENryhHtr4/+b/Nx3plHzLaVgScKpDX8 id
The key's randomart image is:
+---[RSA 4096]----+
|        .        |
|       + o .     |
|      + = + .    |
|       =.* o .   |
|      .oS.+ E  .o|
|      o..o + . ++|
|     . .+ . . + o|
|      .o++ o o ++|
|      o++B=.=o+o+|
root@photon-39BgfUQRO [ ~ ]# eval $(ssh-agent)
Agent pid 666
root@photon-39BgfUQRO [ ~ ]# ssh-add ~/.ssh/id_rsa
Identity added: /root/.ssh/id_rsa (/root/.ssh/id_rsa)
root@photon-39BgfUQRO [ ~ ]#

Step 7. Roll out Kubernetes using kube-up

We are now ready to deploy K8S. Change directory to the Kubernetes extracted folder, and then run the following command. You need KUBERNETES_PROVIDER set to vsphere, and the kube-up.sh script is in the cluster sub-folder. This is all on the same command line by the way (it is wrapped here just for neatness).

root@photon-39BgfUQRO [ ~/kubernetes ]# KUBERNETES_PROVIDER=vsphere \
... Starting cluster using provider: vsphere
... calling verify-prereqs
... calling kube-up

I am not going to reproduce all the output here, but what you should observe is a master VM and 4 minion VMs getting deployed.

k8s-on-vsphere-2You will also see references to K8S being configured via “Salt”, or SaltStack. Salt is a Python-based open-source configuration management software and remote execution engine. Supporting the “Infrastructure as Code” approach to deployment and cloud management, it competes primarily with Puppet, Chef, and Ansible.

If the deployment is successful, you should observer the final output as follows:

Found 4 node(s).
NAME                  STATUS    AGE
kubernetes-minion-1   Ready     3m
kubernetes-minion-2   Ready     3m
kubernetes-minion-3   Ready     3m
kubernetes-minion-4   Ready     3m
Validate output:
NAME                 STATUS    MESSAGE              ERROR
scheduler            Healthy   ok
controller-manager   Healthy   ok
etcd-1               Healthy   {"health": "true"}
etcd-0               Healthy   {"health": "true"}
Cluster validation succeeded
Done, listing cluster services:

Kubernetes master is running at

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

Step 8. Launch the Kubernetes UI

As shown above, you should now be able to connect to the master on whatever IP address is reported in the output. If you append /ui to the URL, and login as “admin”, you should see something like this shown below. To get the admin password, you can find it in the ~/.kube/config file:

root@photon-39BgfUQRO [ ~/kubernetes ]# cat ~/.kube/config | grep password

k8-uiAnd there you have it – Kubernetes running on vSphere. In an upcoming post, I’ll include a useful demo which will demonstrate some of K8S features when running on vSphere, especially around persistent and dynamic volumes. But for now, you can hand this off to your developers to get started with K8S.

To shutdown K8S and remove the VMs, simply run “kube-down.sh”:

root@photon-39BgfUQRO [ ~/kubernetes ]# KUBERNETES_PROVIDER=vsphere \

Further reading

There is some additional reading on deploying Kubernetes with vSphere here.

5 Replies to “Kubernetes on vSphere”

Comments are closed.