Kubernetes on vSphere with kubernetes-anywhere

I already described how you can get started with Kubernetes natively on vSphere using the kube-up/kube-down mechanism. This was pretty straight-forward, but not ideal as it was not very reliable or easy to follow. Since writing that piece, Kubernetes have moved on to a new deployment mechanism called kubernetes-anywhere. In this post, I will show you how to deploy Kubernetes onto a vSphere environment with a vSAN datastore, using the kubernetes-anywhere utility. All of this is done from a Photon OS VM. Now in my previous example, I used the Photon OS OVA, which is a trimmed down version  of the OS. In this example, I have deployed my Photon OS VM using the full ISO image, meaning that all of the tooling I might need (git, awk, etc) are already included. This just saves me a little time. If you want to use the cut-down distribution, check out how to add the additional tooling to the OS with tdnf in my previous post. Now, I have broken up this deployment into a number of distinct parts, but in a nutshell, the steps can be summarized as follows:

  1. Setup guest from where deployment will be done, e.g. Photon OS distro
  2. Download OVA for K8S
  3. Download “kubernetes-anywhere”
  4. Create build environment
  5. Make configuration file for deploying K8S on vSphere
  6. Deploy K8S on vSphere
  7. Get access to the K8S dashboard

Let’s look at these steps in more detail. If you do not need all of this detail, there is an excellent write-up on how to get started with K8S on vSphere found here.

Part 1: Deploy a Photon OS VM with full ISO image

There were only a few things I needed to do with the full ISO image. One was to give it more disk space, for cloning and building. I allocated a 100GB VMDK during VM creation. The next thing I did was to login on the console and enable root logins over SSH, and the final item was to enable and start docker. That’s it. Of course, you can also use many other distros if you wish, but I’m getting to like Photon OS for this sort of testing.


Part 2: Download Photon OS image for VMs to be used by Kubernetes

You can download the OVA from here, or you can deploy it directly from the vSphere Web Client:

Make sure it is on the same vSphere environment where you plan to run K8S. This OVA is another Photon OS image, and will be used to deploy master and node VMs to run Kubernetes. Do not change the name, and do not power it on.


Part 3: git clone kubernetes-anywhere

Login in to your Photon OS VM, install git with “tdnf install git” and then run a git clone of kubernetes-anywhere.

root@photon [ ~ ]# git clone https://github.com/kubernetes/kubernetes-anywhere
 Cloning into 'kubernetes-anywhere'...
 remote: Counting objects: 4092, done.
 remote: Total 4092 (delta 0), reused 0 (delta 0), pack-reused 4092
 Receiving objects: 100% (4092/4092), 3.97 MiB | 642.00 KiB/s, done.
 Resolving deltas: 100% (2573/2573), done.
 Checking connectivity... done.


Part 4: make docker-dev

Now we build a new container environment for the deployment. This is why docker must be enabled and started on the Photon OS VM. This step takes a long time, and generates a lot of output. I’ve put a few snippets here, so you know what to expect. Although it takes a long time to complete the very first time it is run, it is only a few seconds subsequently to make new environments since the container images have all been downloaded.

root@photon [ ~ ]# cd kubernetes-anywhere
root@photon [ ~/kubernetes-anywhere ]# make docker-dev
docker build -t kubernetes-anywhere:v0.0.1 .
Sending build context to Docker daemon 205.3 kB
Step 1 : FROM mhart/alpine-node:6.4.0
6.4.0: Pulling from mhart/alpine-node
e110a4a17941: Pull complete
d6d25f5b0348: Pull complete
Digest: sha256:0a7f08961c8dfaf42910a5aa58549c840bb93fdafab5af3746c11f1cd4062bda
Status: Downloaded newer image for mhart/alpine-node:6.4.0
---> ecd37ad77c2b
Step 2 : RUN apk add --update bash
---> Running in f50126adb9b2
fetch http://dl-cdn.alpinelinux.org/alpine/v3.4/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.4/community/x86_64/APKINDEX.tar.gz
(1/5) Installing ncurses-terminfo-base (6.0-r7)
(2/5) Installing ncurses-terminfo (6.0-r7)
(3/5) Installing ncurses-libs (6.0-r7)
(4/5) Installing readline (6.3.008-r4)
(5/5) Installing bash (4.3.42-r5)
Executing bash-4.3.42-r5.post-install
Executing busybox-1.24.2-r9.trigger
OK: 14 MiB in 18 packages
---> f06561950212
Removing intermediate container f50126adb9b2
Step 3 : ADD ./util/docker-build.sh /opt/
---> f37f66f9fcfe
Removing intermediate container 1da43e7f98a1
Step 4 : RUN /opt/docker-build.sh
---> Running in c8b41a59eb47
fetch http://dl-cdn.alpinelinux.org/alpine/v3.4/main/x86_64/APKINDEX.tar.gz
+ apk add --update git build-base wget curl jq autoconf automake pkgconfig ncurses-dev libtool gperf flex bison ca-certificates
fetch http://dl-cdn.alpinelinux.org/alpine/v3.4/community/x86_64/APKINDEX.tar.gz
(1/38) Upgrading musl (1.1.14-r11 -> 1.1.14-r14).

81300K .......... .......... .......... .......... .......... 99%  203K 0s
81350K .........                                             100%  164M=3m21s
2017-02-15 13:35:14 (405 KB/s) - '/usr/local/bin/kubectl' saved [83312488/83312488]
+ chmod +x /usr/local/bin/kubectl
+ cd /tmp
+ git clone https://github.com/google/jsonnet.git
Cloning into 'jsonnet'..

Saving to: 'terraform_0.7.2_linux_amd64.zip'
     0K .......... .......... .......... .......... ..........  0%  987K 16s
2017-02-15 13:36:11 (8.28 MB/s) - 'terraform_0.7.2_linux_amd64.zip' saved [16332841/16332841]
+ wget https://releases.hashicorp.com/terraform/0.7.2/terraform_0.7.2_SHA256SUMS
--2017-02-15 13:36:11--  https://releases.hashicorp.com/terraform/0.7.2/terraform_0.7.2_SHA256SUMS
Removing intermediate container c3e8ad7bf718
Step 5 : WORKDIR /opt/kubernetes-anywhere
---> Running in 4d419a2d773f
---> 368033c43cdd
Removing intermediate container 4d419a2d773f
Step 6 : ADD . /opt/kubernetes-anywhere/
---> 6d16fcb2ed3d
Removing intermediate container eb77d6546eeb
Step 7 : CMD make
---> Running in c00e6d5d1c94
---> 32497ffce687
Removing intermediate container c00e6d5d1c94
Successfully built 32497ffce687
Starting Kuberetes Anywhere deployment shell in a container
docker run -it --rm --env="PS1=[container]:\w> " --net=host --volume="`pwd`:/opt/kubernetes-anywhere" kubernetes-anywhere:v0.0.1 /bin/bash


Part 5: make config – Configure the vSphere cloud provider for Kubernetes

This is where you tell “kubernetes-anywhere” that you are deploying against a vSphere environment, and this is where you will fill in the details for the vSphere environment. You will need the IP/FQDN of your vCenter server, plus credentials. You will also need datacenter name, cluster name and datastore name. As you can see from this setup, I an deploying onto a vSAN datastore. The other major difference from the defaults is the phase 2 installer container. Do not pick the default, but instead use the one that we have highlighted here. I have highlighted in red where you will need to change from the default, but obviously these are taken from my setup, so you will have to modify to match your own. In phase 3, the addons, Y (for Yes) is the default, so you can just hit return.

[container]:/opt/kubernetes-anywhere> make config

CONFIG_="." kconfig-conf Kconfig
* Kubernetes Minimal Turnup Configuration
* Phase 1: Cluster Resource Provisioning
number of nodes (phase1.num_nodes) [4]
cluster name (phase1.cluster_name) [kubernetes]
* cloud provider: gce, azure or vsphere
cloud provider: gce, azure or vsphere (phase1.cloud_provider) [vsphere]
  * vSphere configuration
  vCenter URL Ex: or myvcenter.io (without https://) (phase1.vSphere.url) [vcsa-06.rainpole.com]
  vCenter port (phase1.vSphere.port) [443]
  vCenter username (phase1.vSphere.username) [administrator@vsphere.local]
  vCenter password (phase1.vSphere.password) [xxxxxxx]
  Does host use self-signed cert (phase1.vSphere.insecure) [true]
  Datacenter (phase1.vSphere.datacenter) [Datacenter]
  Datastore (phase1.vSphere.datastore) [vsanDatastore]
  Specify a valid Cluster, Host or Resource Pool in which to deploy Kubernetes VMs. (phase1.vSphere.resourcepool) [Cluster]
  Number of vCPUs for each VM (phase1.vSphere.vcpu) [1]
  Memory for VM (phase1.vSphere.memory) [2048]
  Name of the template VM imported from OVA (phase1.vSphere.template) [KubernetesAnywhereTemplatePhotonOS.ova]
  Flannel Network (phase1.vSphere.flannel_net) []
* Phase 2: Node Bootstrapping
installer container (phase2.installer_container) [docker.io/ashivani/k8s-ignition:v4]
docker registry (phase2.docker_registry) [gcr.io/google-containers]
kubernetes version (phase2.kubernetes_version) [v1.4.8]
bootstrap provider (phase2.provider) [ignition]
* Phase 3: Deploying Addons
Run the addon manager? (phase3.run_addons) [Y/n/?]
  Run kube-proxy? (phase3.kube_proxy) [Y/n/?]
  Run the dashboard? (phase3.dashboard) [Y/n/?]
  Run heapster? (phase3.heapster) [Y/n/?]
  Run kube-dns? (phase3.kube_dns) [Y/n/?]
# configuration written to .config
make: '.config' is up to date.


Part 6: make deploy – Roll out Kubernetes

This is the last step to get K8S deployed. This rolls out the master and node VMs for Kubernetes, and builds the framework. Again, I have included some snippets, but this does take a little time as well, especially the master (which I’ve seen take between 12min and 15min).

[container]:/opt/kubernetes-anywhere> make deploy
util/config_to_json .config > .config.json
make do WHAT=deploy-cluster
make[1]: Entering directory '/opt/kubernetes-anywhere'
( cd "phase1/$(jq -r '.phase1.cloud_provider' .config.json)"; ./do deploy-cluster )
data.template_file.cloudprovider: Refreshing state...
tls_private_key.kubernetes-root: Creating...
tls_private_key.kubernetes-admin: Creating...
tls_private_key.kubernetes-node: Creating...
vsphere_folder.cluster_folder: Creating...
  datacenter:    "" => "Datacenter"
  existing_path: "" => "<computed>"
  path:          "" => "kubernetes"
tls_private_key.kubernetes-master: Creating...
vsphere_folder.cluster_folder: Creation complete
vsphere_virtual_machine.kubevm2: Creating...
  datacenter:                             "" => "Datacenter"
  disk.#:                                 "" => "1"
  disk.705700330.bootable:                "" => "true"
  disk.705700330.controller_type:         "" => "scsi"
  disk.705700330.datastore:               "" => "vsanDatastore"
  disk.705700330.iops:                    "" => ""
  disk.705700330.keep_on_remove:          "" => ""
  disk.705700330.key:                     "" => "<computed>"
  disk.705700330.name:                    "" => ""
  disk.705700330.size:                    "" => ""
  disk.705700330.template:                "" => "KubernetesAnywhereTemplatePhotonOS.ova"
  disk.705700330.type:                    "" => "thin"
  disk.705700330.uuid:                    "" => "<computed>"
  disk.705700330.vmdk:                    "" => ""
  domain:                                 "" => "vsphere.local"
  enable_disk_uuid:                       "" => "true"
  folder:                                 "" => "kubernetes"
  linked_clone:                           "" => "false"
  memory:                                 "" => "2048"
  memory_reservation:                     "" => "0"
  name:                                   "" => "node1"
  network_interface.#:                    "" => "1"
  network_interface.0.ip_address:         "" => "<computed>"
  network_interface.0.ipv4_address:       "" => "<computed>"
  network_interface.0.ipv4_gateway:       "" => "<computed>"
  network_interface.0.ipv4_prefix_length: "" => "<computed>"
  network_interface.0.ipv6_address:       "" => "<computed>"
  network_interface.0.ipv6_gateway:       "" => "<computed>"
  network_interface.0.ipv6_prefix_length: "" => "<computed>"
  network_interface.0.label:              "" => "VM Network"
  network_interface.0.mac_address:        "" => "<computed>"
  network_interface.0.subnet_mask:        "" => "<computed>"
  resource_pool:                          "" => "Cluster"
  skip_customization:                     "" => "true"
  time_zone:                              "" => "Etc/UTC"
  uuid:                                   "" => "<computed>"
  vcpu:                                   "" => "1"
vsphere_virtual_machine.kubevm4: Creating...
vsphere_virtual_machine.kubevm1: Creating...
vsphere_virtual_machine.kubevm5: Creating...
vsphere_virtual_machine.kubevm3: Creating...
tls_private_key.kubernetes-root: Creation complete
tls_self_signed_cert.kubernetes-root: Creating...
tls_self_signed_cert.kubernetes-root: Creation complete
tls_private_key.kubernetes-node: Creation complete
data.tls_cert_request.kubernetes-node: Refreshing state...
tls_private_key.kubernetes-master: Creation complete
tls_locally_signed_cert.kubernetes-node: Creating...
tls_locally_signed_cert.kubernetes-node: Creation complete
tls_private_key.kubernetes-admin: Creation complete
data.tls_cert_request.kubernetes-admin: Refreshing state...
tls_locally_signed_cert.kubernetes-admin: Creating...
tls_locally_signed_cert.kubernetes-admin: Creation complete
vsphere_virtual_machine.kubevm2: Still creating... (10s elapsed)
vsphere_virtual_machine.kubevm4: Still creating... (10s elapsed)
vsphere_virtual_machine.kubevm1: Still creating... (10s elapsed)
vsphere_virtual_machine.kubevm5: Still creating... (10s elapsed)
vsphere_virtual_machine.kubevm3: Still creating... (10s elapsed)
null_resource.master: Still creating... (14m0s elapsed)
null_resource.master (remote-exec):  96 75.8M   96 73.5M    0     0  94543      0  0:14:01  0:13:35  0:00:26  277k
null_resource.master (remote-exec):  97 75.8M   97 73.7M    0     0  94772      0  0:13:59  0:13:36  0:00:23  278k
null_resource.master (remote-exec):  97 75.8M   97 73.9M    0     0  94856      0  0:13:58  0:13:37  0:00:21  254k
null_resource.master (remote-exec):  97 75.8M   97 74.1M    0     0  94981      0  0:13:57  0:13:38  0:00:19  233k
null_resource.master (remote-exec):  97 75.8M   97 74.3M    0     0  95093      0  0:13:56  0:13:39  0:00:17  215k
null_resource.master (remote-exec):  98 75.8M   98 74.5M    0     0  95234      0  0:13:55  0:13:40  0:00:15  203k
null_resource.master (remote-exec):  98 75.8M   98 74.6M    0     0  95355      0  0:13:54  0:13:41  0:00:13  185k
null_resource.master (remote-exec):  98 75.8M   98 74.8M    0     0  95501      0  0:13:52  0:13:42  0:00:10  196k
null_resource.master (remote-exec):  98 75.8M   98 75.1M    0     0  95639      0  0:13:51  0:13:43  0:00:08  196k
null_resource.master (remote-exec):  99 75.8M   99 75.2M    0     0  95727      0  0:13:51  0:13:44  0:00:07  194k
null_resource.master: Still creating... (14m10s elapsed)
null_resource.master (remote-exec):  99 75.8M   99 75.3M    0     0  95782      0  0:13:50  0:13:45  0:00:05  181k
null_resource.master (remote-exec):  99 75.8M   99 75.5M    0     0  95844      0  0:13:50  0:13:46  0:00:04  170k
null_resource.master (remote-exec):  99 75.8M   99 75.6M    0     0  95903      0  0:13:49  0:13:47  0:00:02  157k
null_resource.master (remote-exec):  99 75.8M   99 75.8M    0     0  95969      0  0:13:48  0:13:48 --:--:--  148k
null_resource.master (remote-exec): 100 75.8M  100 75.8M    0     0  95991      0  0:13:48  0:13:48 --:--:--  141k
null_resource.master: Provisioning with 'local-exec'...
Apply complete! Resources: 19 added, 0 changed, 0 destroyed.

The state of your infrastructure has been saved to the path
below. This state is required to modify and destroy your
infrastructure, so keep it safe. To inspect the complete state
use the `terraform show` command.

State path: ./.tmp/terraform.tfstate
make[1]: Leaving directory '/opt/kubernetes-anywhere'
KUBECONFIG="$(pwd)/phase1/$(jq -r '.phase1.cloud_provider' .config.json)/.tmp/kubeconfig.json" ./util/validate
Validation: Expected 5 healthy nodes; found 0. (10s elapsed)
Validation: Expected 5 healthy nodes; found 0. (20s elapsed)
Validation: Expected 5 healthy nodes; found 5. (30s elapsed)
Validation: Success!
KUBECONFIG="$(pwd)/phase1/$(jq -r '.phase1.cloud_provider' .config.json)/.tmp/kubeconfig.json" ./phase3/do deploy
+ case "${1:-}" in
+ deploy
+ gen
+ cd ./phase3
+ mkdir -p .tmp/
+ jsonnet --multi .tmp/ --tla-code-file cfg=../.config.json all.jsonnet
+ mkdir -p /tmp/kubectl/
+ HOME=/tmp/kubectl
+ kubectl apply -f ./.tmp/
deployment "kubernetes-dashboard" created
service "kubernetes-dashboard" created
deployment "heapster-v1.2.0" created
service "heapster" created
replicationcontroller "kube-dns-v19" created
service "kube-dns" created
daemonset "kube-proxy" created

When this step completes, the master and nodes VMs should now be visible in the vSphere web client, in a folder called kubernetes:

Part 7 – Get access to the K8S dashboard

The kubectl commands will allow you to figure out which node and port are being used to run the dashboard. You can then open a browser and point to it. The first step is to download “kubectl”, and then point it at your K8S configuration.

[container]:/opt/kubernetes-anywhere> curl -LO https://storage.googleapis.com/kubernetes-release/release/v1.4.8/bin/linux/amd64/kubectl

[container]:/opt/kubernetes-anywhere> chmod u+x kubectl

[container]:/opt/kubernetes-anywhere> export KUBECONFIG=phase1/vsphere/.tmp/kubeconfig.json

[container]:/opt/kubernetes-anywhere> kubectl describe service kubernetes-dashboard --namespace=kube-system| grep -i NodePort 
 Type:            NodePort NodePort:        <unset>    31402/TCP

[container]:/opt/kubernetes-anywhere> kubectl get pods --namespace=kube-system| grep -i dashboard
 kubernetes-dashboard-1763797262-abexo   1/1       Running   0          13m

[container]:/opt/kubernetes-anywhere> kubectl describe pod kubernetes-dashboard-1763797262-abexo --namespace=kube-system| grep Node
 Node:        node4/

Now if I point my browser at, I can launch my K8S dashboard.

And there you have it – K8S deployed on vSphere with just a handful of steps. Very seamless. Kudos to our vSphere Cloud Provider team who worked very hard to make this happen.

12 Replies to “Kubernetes on vSphere with kubernetes-anywhere”

  1. great tutorial!
    can you please explain what means

    resource_pool: “” => “Cluster”

    it didn’t work for me, instead i’ve specified esxi host ip

    1. You can group multiple hosts into a cluster. All hosts contribute to the resource pool.

      You should still be able to select a cluster if you have created it and added your host to it.

      In your case, it seems like you may not have a cluster, but just a single host. In this case, the single host is your resource pool.

      One can also create additional resource pools under cluster for multi-tenancy, each k8s deployment using its own allocated set of resources.

  2. Hello Cormac, thank you for the great guide! I was able to get through 90% of it, however when I run make deploy I get this error on the master server only. The other nodes seem to create just fine. Any ideas? I am totally new to K8S so I am getting to a point where I have no idea where to go from here.

    null_resource.master (remote-exec): v1.5.3: Pulling from google-containers/hyperkube-amd64
    null_resource.master (remote-exec): Digest: sha256:1c44cdd9526667e5b11b51151940824218a9ca97c6276f8f29830895907147cd
    null_resource.master (remote-exec): Status: Image is up to date for gcr.io/google-containers/hyperkube-amd64:v1.5.3
    null_resource.master (remote-exec): Unable to find image ‘remarkable/k8s-ignition:v1’ locally
    null_resource.master (remote-exec): Pulling repository docker.io/remarkable/k8s-ignition
    Error applying plan:

  3. Thanks Cormac, I actually figured it out. I did not set the DHCP scope on the Kubernetes VLAN I created, so once I assigned the nodes a static IP, and enabled SSH on all of them, I was able to get the deploy to succeed. Thanks for the reply, and thanks again for the guide!

  4. Thanks Cormac, Have you ever tried to run the “make deploy” and fail right the way after the folder is being created? I cannot even see the VM cloning tasks start and the error compalints:

    Error applying plan:

    5 error(s) occurred:

    * vsphere_virtual_machine.kubevm2: vm ‘KubernetesAnywhereTemplatePhotonOS.ova’ not found
    * vsphere_virtual_machine.kubevm5: vm ‘KubernetesAnywhereTemplatePhotonOS.ova’ not found
    * vsphere_virtual_machine.kubevm3: vm ‘KubernetesAnywhereTemplatePhotonOS.ova’ not found
    * vsphere_virtual_machine.kubevm4: vm ‘KubernetesAnywhereTemplatePhotonOS.ova’ not found
    * vsphere_virtual_machine.kubevm1: vm ‘KubernetesAnywhereTemplatePhotonOS.ova’ not found

    Terraform does not automatically rollback in the face of errors.
    Instead, your Terraform state file has been partially updated with
    any resources that successfully completed. Please address the error
    above and apply again to incrementally change your infrastructure.
    Makefile:63: recipe for target ‘do’ failed
    make[1]: *** [do] Error 1
    make[1]: Leaving directory ‘/opt/kubernetes-anywhere’
    Makefile:41: recipe for target ‘deploy-cluster’ failed
    make: *** [deploy-cluster] Error 2

    1. Did you happen to change the name of the OVA when you deployed it in step 2? If so, you need to provide the exact name during the “make config” step.

  5. I confirm that I’m also having this issue

    * vsphere_virtual_machine.kubevm5: vm ‘KubernetesAnywhereTemplatePhotonOS.ova’ not found
    * vsphere_virtual_machine.kubevm3: vm ‘KubernetesAnywhereTemplatePhotonOS.ova’ not found
    * vsphere_virtual_machine.kubevm1: vm ‘KubernetesAnywhereTemplatePhotonOS.ova’ not found
    * vsphere_virtual_machine.kubevm2: vm ‘KubernetesAnywhereTemplatePhotonOS.ova’ not found
    * vsphere_virtual_machine.kubevm4: vm ‘KubernetesAnywhereTemplatePhotonOS.ova’ not found

    I’ve removed OVA, placed it again, no difference
    Do you think i need to raise issue on github?

Comments are closed.