Kubernetes on vSphere with kubernetes-anywhere

I already described how you can get started with Kubernetes natively on vSphere using the kube-up/kube-down mechanism. This was pretty straight-forward, but not ideal as it was not very reliable or easy to follow. Since writing that piece, Kubernetes have moved on to a new deployment mechanism called kubernetes-anywhere. In this post, I will show you how to deploy Kubernetes onto a vSphere environment with a vSAN datastore, using the kubernetes-anywhere utility. All of this is done from a Photon OS VM. Now in my previous example, I used the Photon OS OVA, which is a trimmed down version  of the OS. In this example, I have deployed my Photon OS VM using the full ISO image, meaning that all of the tooling I might need (git, awk, etc) are already included. This just saves me a little time. If you want to use the cut-down distribution, check out how to add the additional tooling to the OS with tdnf in my previous post. Now, I have broken up this deployment into a number of distinct parts, but in a nutshell, the steps can be summarized as follows:

1. Setup guest from where deployment will be done, e.g. Photon OS distro
2. Download OVA for K8S
3. Download “kubernetes-anywhere”
4. Create build environment
5. Make configuration file for deploying K8S on vSphere
6. Deploy K8S on vSphere
7. Get access to the K8S dashboard

Let’s look at these steps in more detail. If you do not need all of this detail, there is an excellent write-up on how to get started with K8S on vSphere found here.

Part 1: Deploy a Photon OS VM with full ISO image

There were only a few things I needed to do with the full ISO image. One was to give it more disk space, for cloning and building. I allocated a 100GB VMDK during VM creation. The next thing I did was to login on the console and enable root logins over SSH, and the final item was to enable and start docker. That’s it. Of course, you can also use many other distros if you wish, but I’m getting to like Photon OS for this sort of testing.

Part 2: Download Photon OS image for VMs to be used by Kubernetes

You can download the OVA from here, or you can deploy it directly from the vSphere Web Client:

Make sure it is on the same vSphere environment where you plan to run K8S. This OVA is another Photon OS image, and will be used to deploy master and node VMs to run Kubernetes. Do not change the name, and do not power it on.

Part 3: git clone kubernetes-anywhere

Login in to your Photon OS VM, install git with “tdnf install git” and then run a git clone of kubernetes-anywhere.

root@photon [ ~ ]# git clone https://github.com/kubernetes/kubernetes-anywhere
 Cloning into 'kubernetes-anywhere'...
 remote: Counting objects: 4092, done.
 remote: Total 4092 (delta 0), reused 0 (delta 0), pack-reused 4092
 Receiving objects: 100% (4092/4092), 3.97 MiB | 642.00 KiB/s, done.
 Resolving deltas: 100% (2573/2573), done.
 Checking connectivity... done.

Part 4: make docker-dev

Now we build a new container environment for the deployment. This is why docker must be enabled and started on the Photon OS VM. This step takes a long time, and generates a lot of output. I’ve put a few snippets here, so you know what to expect. Although it takes a long time to complete the very first time it is run, it is only a few seconds subsequently to make new environments since the container images have all been downloaded.

root@photon [ ~ ]# cd kubernetes-anywhere
root@photon [ ~/kubernetes-anywhere ]# make docker-dev
docker build -t kubernetes-anywhere:v0.0.1 .
Sending build context to Docker daemon 205.3 kB
Step 1 : FROM mhart/alpine-node:6.4.0
6.4.0: Pulling from mhart/alpine-node
e110a4a17941: Pull complete
d6d25f5b0348: Pull complete
Digest: sha256:0a7f08961c8dfaf42910a5aa58549c840bb93fdafab5af3746c11f1cd4062bda
Status: Downloaded newer image for mhart/alpine-node:6.4.0
---> ecd37ad77c2b
Step 2 : RUN apk add --update bash
---> Running in f50126adb9b2
fetch http://dl-cdn.alpinelinux.org/alpine/v3.4/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.4/community/x86_64/APKINDEX.tar.gz
(1/5) Installing ncurses-terminfo-base (6.0-r7)
(2/5) Installing ncurses-terminfo (6.0-r7)
(3/5) Installing ncurses-libs (6.0-r7)
(4/5) Installing readline (6.3.008-r4)
(5/5) Installing bash (4.3.42-r5)
Executing bash-4.3.42-r5.post-install
Executing busybox-1.24.2-r9.trigger
OK: 14 MiB in 18 packages
---> f06561950212
Removing intermediate container f50126adb9b2
Step 3 : ADD ./util/docker-build.sh /opt/
---> f37f66f9fcfe
Removing intermediate container 1da43e7f98a1
Step 4 : RUN /opt/docker-build.sh
---> Running in c8b41a59eb47
fetch http://dl-cdn.alpinelinux.org/alpine/v3.4/main/x86_64/APKINDEX.tar.gz
+ apk add --update git build-base wget curl jq autoconf automake pkgconfig ncurses-dev libtool gperf flex bison ca-certificates
fetch http://dl-cdn.alpinelinux.org/alpine/v3.4/community/x86_64/APKINDEX.tar.gz
(1/38) Upgrading musl (1.1.14-r11 -> 1.1.14-r14).

81300K .......... .......... .......... .......... .......... 99%  203K 0s
81350K .........                                             100%  164M=3m21s
2017-02-15 13:35:14 (405 KB/s) - '/usr/local/bin/kubectl' saved [83312488/83312488]
+ chmod +x /usr/local/bin/kubectl
+ cd /tmp
+ git clone https://github.com/google/jsonnet.git
Cloning into 'jsonnet'..
.

Saving to: 'terraform_0.7.2_linux_amd64.zip'
     0K .......... .......... .......... .......... ..........  0%  987K 16s
.
2017-02-15 13:36:11 (8.28 MB/s) - 'terraform_0.7.2_linux_amd64.zip' saved [16332841/16332841]
+ wget https://releases.hashicorp.com/terraform/0.7.2/terraform_0.7.2_SHA256SUMS
--2017-02-15 13:36:11--  https://releases.hashicorp.com/terraform/0.7.2/terraform_0.7.2_SHA256SUMS
.
Removing intermediate container c3e8ad7bf718
Step 5 : WORKDIR /opt/kubernetes-anywhere
---> Running in 4d419a2d773f
---> 368033c43cdd
Removing intermediate container 4d419a2d773f
Step 6 : ADD . /opt/kubernetes-anywhere/
---> 6d16fcb2ed3d
Removing intermediate container eb77d6546eeb
Step 7 : CMD make
---> Running in c00e6d5d1c94
---> 32497ffce687
Removing intermediate container c00e6d5d1c94
Successfully built 32497ffce687
Starting Kuberetes Anywhere deployment shell in a container
docker run -it --rm --env="PS1=[container]:\w> " --net=host --volume="`pwd`:/opt/kubernetes-anywhere" kubernetes-anywhere:v0.0.1 /bin/bash
[container]:/opt/kubernetes-anywhere>

[container]:/opt/kubernetes-anywhere> make config

CONFIG_="." kconfig-conf Kconfig
*
* Kubernetes Minimal Turnup Configuration
*
*
* Phase 1: Cluster Resource Provisioning
*
number of nodes (phase1.num_nodes) [4]
cluster name (phase1.cluster_name) [kubernetes]
*
* cloud provider: gce, azure or vsphere
*
cloud provider: gce, azure or vsphere (phase1.cloud_provider) [vsphere]
  *
  * vSphere configuration
  *
  vCenter URL Ex: 10.192.10.30 or myvcenter.io (without https://) (phase1.vSphere.url) [vcsa-06.rainpole.com]
  vCenter port (phase1.vSphere.port) [443]
  vCenter username (phase1.vSphere.username) [administrator@vsphere.local]
  vCenter password (phase1.vSphere.password) [xxxxxxx]
  Does host use self-signed cert (phase1.vSphere.insecure) [true]
  Datacenter (phase1.vSphere.datacenter) [Datacenter]
  Datastore (phase1.vSphere.datastore) [vsanDatastore]
  Specify a valid Cluster, Host or Resource Pool in which to deploy Kubernetes VMs. (phase1.vSphere.resourcepool) [Cluster]
  Number of vCPUs for each VM (phase1.vSphere.vcpu) [1]
  Memory for VM (phase1.vSphere.memory) [2048]
  Name of the template VM imported from OVA (phase1.vSphere.template) [KubernetesAnywhereTemplatePhotonOS.ova]
  Flannel Network (phase1.vSphere.flannel_net) [172.1.0.0/16]
*
* Phase 2: Node Bootstrapping
*
installer container (phase2.installer_container) [docker.io/ashivani/k8s-ignition:v4]
docker registry (phase2.docker_registry) [gcr.io/google-containers]
kubernetes version (phase2.kubernetes_version) [v1.4.8]
bootstrap provider (phase2.provider) [ignition]
*
* Phase 3: Deploying Addons
*
Run the addon manager? (phase3.run_addons) [Y/n/?]
  Run kube-proxy? (phase3.kube_proxy) [Y/n/?]
  Run the dashboard? (phase3.dashboard) [Y/n/?]
  Run heapster? (phase3.heapster) [Y/n/?]
  Run kube-dns? (phase3.kube_dns) [Y/n/?]
#
# configuration written to .config
#
make: '.config' is up to date.
[container]:/opt/kubernetes-anywhere>

Part 6: make deploy – Roll out Kubernetes

This is the last step to get K8S deployed. This rolls out the master and node VMs for Kubernetes, and builds the framework. Again, I have included some snippets, but this does take a little time as well, especially the master (which I’ve seen take between 12min and 15min).

[container]:/opt/kubernetes-anywhere> make deploy
util/config_to_json .config > .config.json
make do WHAT=deploy-cluster
make[1]: Entering directory '/opt/kubernetes-anywhere'
( cd "phase1/$(jq -r '.phase1.cloud_provider' .config.json)"; ./do deploy-cluster )
.tmp/vSphere-kubernetes.tf
data.template_file.cloudprovider: Refreshing state...
tls_private_key.kubernetes-root: Creating...
 .
tls_private_key.kubernetes-admin: Creating...
 .
tls_private_key.kubernetes-node: Creating...
 .
vsphere_folder.cluster_folder: Creating...
  datacenter:    "" => "Datacenter"
  existing_path: "" => "<computed>"
  path:          "" => "kubernetes"
tls_private_key.kubernetes-master: Creating...
 .
vsphere_folder.cluster_folder: Creation complete
vsphere_virtual_machine.kubevm2: Creating...
  datacenter:                             "" => "Datacenter"
  disk.#:                                 "" => "1"
  disk.705700330.bootable:                "" => "true"
  disk.705700330.controller_type:         "" => "scsi"
  disk.705700330.datastore:               "" => "vsanDatastore"
  disk.705700330.iops:                    "" => ""
  disk.705700330.keep_on_remove:          "" => ""
  disk.705700330.key:                     "" => "<computed>"
  disk.705700330.name:                    "" => ""
  disk.705700330.size:                    "" => ""
  disk.705700330.template:                "" => "KubernetesAnywhereTemplatePhotonOS.ova"
  disk.705700330.type:                    "" => "thin"
  disk.705700330.uuid:                    "" => "<computed>"
  disk.705700330.vmdk:                    "" => ""
  domain:                                 "" => "vsphere.local"
  enable_disk_uuid:                       "" => "true"
  folder:                                 "" => "kubernetes"
  linked_clone:                           "" => "false"
  memory:                                 "" => "2048"
  memory_reservation:                     "" => "0"
  name:                                   "" => "node1"
  network_interface.#:                    "" => "1"
  network_interface.0.ip_address:         "" => "<computed>"
  network_interface.0.ipv4_address:       "" => "<computed>"
  network_interface.0.ipv4_gateway:       "" => "<computed>"
  network_interface.0.ipv4_prefix_length: "" => "<computed>"
  network_interface.0.ipv6_address:       "" => "<computed>"
  network_interface.0.ipv6_gateway:       "" => "<computed>"
  network_interface.0.ipv6_prefix_length: "" => "<computed>"
  network_interface.0.label:              "" => "VM Network"
  network_interface.0.mac_address:        "" => "<computed>"
  network_interface.0.subnet_mask:        "" => "<computed>"
  resource_pool:                          "" => "Cluster"
  skip_customization:                     "" => "true"
  time_zone:                              "" => "Etc/UTC"
  uuid:                                   "" => "<computed>"
  vcpu:                                   "" => "1"
vsphere_virtual_machine.kubevm4: Creating...
.
vsphere_virtual_machine.kubevm1: Creating...
.
vsphere_virtual_machine.kubevm5: Creating...
.
vsphere_virtual_machine.kubevm3: Creating...
.
tls_private_key.kubernetes-root: Creation complete
tls_self_signed_cert.kubernetes-root: Creating...
.
tls_self_signed_cert.kubernetes-root: Creation complete
tls_private_key.kubernetes-node: Creation complete
data.tls_cert_request.kubernetes-node: Refreshing state...
tls_private_key.kubernetes-master: Creation complete
tls_locally_signed_cert.kubernetes-node: Creating...
.
tls_locally_signed_cert.kubernetes-node: Creation complete
tls_private_key.kubernetes-admin: Creation complete
data.tls_cert_request.kubernetes-admin: Refreshing state...
tls_locally_signed_cert.kubernetes-admin: Creating...
.
tls_locally_signed_cert.kubernetes-admin: Creation complete
vsphere_virtual_machine.kubevm2: Still creating... (10s elapsed)
vsphere_virtual_machine.kubevm4: Still creating... (10s elapsed)
vsphere_virtual_machine.kubevm1: Still creating... (10s elapsed)
vsphere_virtual_machine.kubevm5: Still creating... (10s elapsed)
vsphere_virtual_machine.kubevm3: Still creating... (10s elapsed)
.
null_resource.master: Still creating... (14m0s elapsed)
null_resource.master (remote-exec):  96 75.8M   96 73.5M    0     0  94543      0  0:14:01  0:13:35  0:00:26  277k
null_resource.master (remote-exec):  97 75.8M   97 73.7M    0     0  94772      0  0:13:59  0:13:36  0:00:23  278k
null_resource.master (remote-exec):  97 75.8M   97 73.9M    0     0  94856      0  0:13:58  0:13:37  0:00:21  254k
null_resource.master (remote-exec):  97 75.8M   97 74.1M    0     0  94981      0  0:13:57  0:13:38  0:00:19  233k
null_resource.master (remote-exec):  97 75.8M   97 74.3M    0     0  95093      0  0:13:56  0:13:39  0:00:17  215k
null_resource.master (remote-exec):  98 75.8M   98 74.5M    0     0  95234      0  0:13:55  0:13:40  0:00:15  203k
null_resource.master (remote-exec):  98 75.8M   98 74.6M    0     0  95355      0  0:13:54  0:13:41  0:00:13  185k
null_resource.master (remote-exec):  98 75.8M   98 74.8M    0     0  95501      0  0:13:52  0:13:42  0:00:10  196k
null_resource.master (remote-exec):  98 75.8M   98 75.1M    0     0  95639      0  0:13:51  0:13:43  0:00:08  196k
null_resource.master (remote-exec):  99 75.8M   99 75.2M    0     0  95727      0  0:13:51  0:13:44  0:00:07  194k
null_resource.master: Still creating... (14m10s elapsed)
null_resource.master (remote-exec):  99 75.8M   99 75.3M    0     0  95782      0  0:13:50  0:13:45  0:00:05  181k
null_resource.master (remote-exec):  99 75.8M   99 75.5M    0     0  95844      0  0:13:50  0:13:46  0:00:04  170k
null_resource.master (remote-exec):  99 75.8M   99 75.6M    0     0  95903      0  0:13:49  0:13:47  0:00:02  157k
null_resource.master (remote-exec):  99 75.8M   99 75.8M    0     0  95969      0  0:13:48  0:13:48 --:--:--  148k
null_resource.master (remote-exec): 100 75.8M  100 75.8M    0     0  95991      0  0:13:48  0:13:48 --:--:--  141k
null_resource.master: Provisioning with 'local-exec'...
. 
Apply complete! Resources: 19 added, 0 changed, 0 destroyed.

The state of your infrastructure has been saved to the path
below. This state is required to modify and destroy your
infrastructure, so keep it safe. To inspect the complete state
use the `terraform show` command.

State path: ./.tmp/terraform.tfstate
make[1]: Leaving directory '/opt/kubernetes-anywhere'
KUBECONFIG="$(pwd)/phase1/$(jq -r '.phase1.cloud_provider' .config.json)/.tmp/kubeconfig.json" ./util/validate
Validation: Expected 5 healthy nodes; found 0. (10s elapsed)
Validation: Expected 5 healthy nodes; found 0. (20s elapsed)
Validation: Expected 5 healthy nodes; found 5. (30s elapsed)
Validation: Success!
KUBECONFIG="$(pwd)/phase1/$(jq -r '.phase1.cloud_provider' .config.json)/.tmp/kubeconfig.json" ./phase3/do deploy
+ case "${1:-}" in
+ deploy
+ gen
+ cd ./phase3
+ mkdir -p .tmp/
+ jsonnet --multi .tmp/ --tla-code-file cfg=../.config.json all.jsonnet
.tmp/dashboard-deployment.json
.tmp/dashboard-svc.json
.tmp/heapster-deployment.json
.tmp/heapster-svc.json
.tmp/kube-dns-deployment.json
.tmp/kube-dns-svc.json
.tmp/kube-proxy.json
+ mkdir -p /tmp/kubectl/
+ HOME=/tmp/kubectl
+ kubectl apply -f ./.tmp/
deployment "kubernetes-dashboard" created
service "kubernetes-dashboard" created
deployment "heapster-v1.2.0" created
service "heapster" created
replicationcontroller "kube-dns-v19" created
service "kube-dns" created
daemonset "kube-proxy" created
[container]:/opt/kubernetes-anywhere>

When this step completes, the master and nodes VMs should now be visible in the vSphere web client, in a folder called kubernetes:

Part 7 – Get access to the K8S dashboard

The kubectl commands will allow you to figure out which node and port are being used to run the dashboard. You can then open a browser and point to it. The first step is to download “kubectl”, and then point it at your K8S configuration.

[container]:/opt/kubernetes-anywhere> curl -LO https://storage.googleapis.com/kubernetes-release/release/v1.4.8/bin/linux/amd64/kubectl

[container]:/opt/kubernetes-anywhere> chmod u+x kubectl

[container]:/opt/kubernetes-anywhere> export KUBECONFIG=phase1/vsphere/.tmp/kubeconfig.json

[container]:/opt/kubernetes-anywhere> kubectl describe service kubernetes-dashboard --namespace=kube-system| grep -i NodePort 
 Type:            NodePort NodePort:        <unset>    31402/TCP

[container]:/opt/kubernetes-anywhere> kubectl get pods --namespace=kube-system| grep -i dashboard
 kubernetes-dashboard-1763797262-abexo   1/1       Running   0          13m

[container]:/opt/kubernetes-anywhere> kubectl describe pod kubernetes-dashboard-1763797262-abexo --namespace=kube-system| grep Node
 Node:        node4/10.27.51.132
 

Now if I point my browser at http://10.27.51.132:31402, I can launch my K8S dashboard.