Kubernetes on Photon Controller

kubernetesAnother container framework that VMware customers can evaluate on Photon Controller is Kubernetes, developed by Google and now open-sourced. Kubernetes is another popular framework that allows customers to automate, manage and scale containers. Just like my previous article on Mesos and Docker Swarm, the Photon Controller and Kubernetes deployment steps are very similar. While I will show the additional steps required to get Kubernetes deployed, I wanted to focus once again on the “what do I do now?” question as this is pretty much the most common question from folks who have gone through the deployment of the Photon Controller and the creation of a container framework/cluster. For this post, I am going to show you how to use the “kubectl” CLI utility, and show how to get started with some K8S containers (K8S is short-hand for Kubernetes).

*** Please note that at the time of writing, Photon Controller is still not GA ***

*** The steps highlighted here may change in the GA version of the product ***

Let’s go over the Photon Controller deployment steps once more. Deploying the Photon Controller Installer OVA, creating the Photon Controller framework and creating the tenants, resource tickets and projects are identical to those outlined in the Docker Swarm post. I won’t cover them here, so please check back to that post for the basic steps.

1. Create a Kubernetes image

However I still need to get a K8S VMDK/VM “image” and enable my Photon Controller deployment for K8S clusters. This image will be used to create the K8S etcd, master and slave worker machines (VMs) which will form the cluster and allow us to deploy containers. Below are the steps to do just that (you’ll find further detail in the previous posts on Swarm and Mesos). The link to the Kubernetes image can be found here. I recommend using the “-i EAGER” option which uploads the image to the image datastore. This will speed up cluster creation as the image is already in-place. As I mentioned in previous posts, you should also wait for the image to get uploaded to the image datastore before proceeding to the next step of building the cluster to avoid potential timeouts.

I am running these Photon CLI commands from my desktop. The Photon CLI utility can be downloaded from github here. I first show that there is no image, then create the image from one I have locally in my “Downloads” folder on my desktop. Then I enable my deployment for Kubernetes.

C:\Users\chogan>photon image list
Using target 'http://10.27.44.34:28080'
ID                                    Name                             State \
 Size(Byte)   Replication_type  ReplicationProgress  SeedingProgress
8e924447-d248-44b3-a811-1cf62e0caf3d  photon-management-vm-disk1.vmdk  READY \
 41943040000  ON_DEMAND         17%                  100%
Total: 1

C:\Users\chogan>photon image create \
Downloads\photon-kubernetes-vm-disk1.vmdk -n k8-vm.vmdk -i EAGER
Using target 'http://10.27.44.34:28080'
Created image 'k8-vm.vmdk' ID: 702b394e-c529-4252-8d3c-324bf0710522

C:\Users\chogan>photon deployment enable-cluster-type \ 
7c2941f4-5f93-495d-843a-693c6106111e -k KUBERNETES \
-i 702b394e-c529-4252-8d3c-324bf0710522
Are you sure [y/n]? y
Using target 'http://10.27.44.34:28080'
Cluster Type: KUBERNETES
Image ID:     702b394e-c529-4252-8d3c-324bf0710522

C:\Users\chogan>photon deployment show 7c2941f4-5f93-495d-843a-693c6106111e
Using target 'http://10.27.44.34:28080'

Deployment ID: 7c2941f4-5f93-495d-843a-693c6106111e
  State:                       READY
  Image Datastores:            [isilion-nfs-01]
  Use image datastore for vms: false
  Syslog Endpoint:             -
  Ntp Endpoint:                10.27.51.252
  LoadBalancer:
    Enabled:                   true
    Address:                   10.27.44.34
 
Auth:
   Enabled:                   false
  
Stats:
   Enabled:                   false
  
Migration status:
    Completed data migration cycles:          0
    Current data migration cycles progress:   0 / 0
    VIB upload progress:                      0 / 0
  
Cluster Configurations:
    ClusterConfiguration 1:
      Kind:      clusterConfig
      Type:      KUBERNETES
      ImageID:   702b394e-c529-4252-8d3c-324bf0710522    

  Job            VM IP(s)     Ports
  CloudStore     10.27.44.34  19000, 19001
  Deployer       10.27.44.34  18000, 18001
  Housekeeper    10.27.44.34  16000, 16001
  LoadBalancer   10.27.44.34  28080, 4343, 443, 80
  ManagementApi  10.27.44.34  9000
  ManagementUi   10.27.44.34  20000, 20001
  RootScheduler  10.27.44.34  13010
  Zookeeper      10.27.44.34  2181, 2888, 3888

  VM IP        Host IP     VM ID                                 VM Name
  10.27.44.34  10.27.51.8  a22bba76-99c4-407b-9d4f-30844d1cf5a3  ec-mgmt-10-27-51-87f27e
C:\Users\chogan>

 2. Create a Kubernetes cluster

Now that we have the Kubernetes image uploaded to the image datastore, and the deployment is configured for K8S, the cluster can be created. I simply called it “Kube”. You will need 2 static IP address for K8S (one for master node and the other for etcd node) on the management network. The management network will also require DHCP for the slave nodes. You will also need a range of IP addresses on the container network (I have picked 10.2.0.0/16 in this example).

C:\Users\chogan>photon cluster create -n Kube -k KUBERNETES \
--dns 10.16.142.110 --gateway 10.27.47.254 --netmask 255.255.240.0 \
--master-ip 10.27.44.35 --container-network 10.2.0.0/16 --etcd1 10.27.44.36 -s 1
Using target 'http://10.27.44.34:28080'
etcd server 2 static IP address (leave blank for none):

Creating cluster: Kube (KUBERNETES)
  Slave count: 1
Are you sure [y/n]? y
Cluster created: ID = 54578909-6317-4d7d-ab80-005a2f0e9f60
Note: the cluster has been created with minimal resources. You can use the cluster now.
A background task is running to gradually expand the cluster to its target capacity.
You can run 'cluster show 54578909-6317-4d7d-ab80-005a2f0e9f60' to see the state of \
the cluster.

C:\Users\chogan>

 At this point, we can examine the cluster. Only the master and etcd nodes are displayed, slave are not displayed:

C:\Users\chogan>photon cluster show 54578909-6317-4d7d-ab80-005a2f0e9f60
Using target 'http://10.27.44.34:28080'
Cluster ID:             54578909-6317-4d7d-ab80-005a2f0e9f60
  Name:                 Kube
  State:                READY
  Type:                 KUBERNETES
  Slave count:          1
  Extended Properties:  map[etcd_ips:10.27.44.36 master_ip:10.27.44.35 \
  gateway:10.27.47.254 container_network:10.2.0.0/16 netmask:255.255.240.0 \
  dns:10.16.142.110]
VM ID                                 VM Name                                      \
VM IP
074d87d6-e259-42e2-a68a-ddb4f4e88afe  master-459e501f-1d17-4a89-b408-a78902fb6db2  \
10.27.44.35
b3093ee6-b02f-4005-b7ab-bed9ef439d0d  etcd-13cf02af-a5f7-47b6-826c-937d018312ad    \
10.27.44.36

To see the slave(s), the following command can be used:

C:\Users\chogan>photon cluster list_vms 54578909-6317-4d7d-ab80-005a2f0e9f60
Using target 'http://10.27.44.34:28080'
ID                                    Name                                         State
074d87d6-e259-42e2-a68a-ddb4f4e88afe  master-459e501f-1d17-4a89-b408-a78902fb6db2  STARTED
18eca5ba-da65-4f8a-b2d4-f2d426b83317  slave-6cf96592-8a00-4365-a6c3-596e1f7706ea   STARTED
b3093ee6-b02f-4005-b7ab-bed9ef439d0d  etcd-13cf02af-a5f7-47b6-826c-937d018312ad    STARTED
Total: 3
STARTED: 3

C:\Users\chogan>

3. Some K8S terminology

Great, we now have the Kubernetes cluster deployed on top of Photon Controller. As mentioned in the introduction, this inevitably leads us on to the next question which is “what do I do next?”. I mentioned that we will do some demoing with the “kubectl” command, a K8S CLI command. You can download “kubectl” from the K8S github repository. Before we get into using “kubectl”, the K8S CLI utility, lets talk a little bit about the new concepts and terminology associated with K8S.

The first term to clarify is nodes. In K8S, these are the worker machines. In our deployment, Photon Controller rolls out VMs for the master and slave node(s). Each node has the necessary components to run what are called Pods.

So what is a Pod? A pod is a term used for a group of one or more containers, the shared storage for those containers, and options about how to run the containers. You could also think of this as a term of applications running on the cluster.

This brings us onto Replica Controllers. The sole purpose of a replica controller is to ensure that a specified number of pod “replicas” are running at any one time to ensure that, even in the event of a failure, the Pods (or applications running in containers) continues to run.

The other term you will come across is Service. Pods are designed to be transient – they can fail/go-away at any time. However certain applications may be composed of what might be termed front-end Pods and back-end Pods. The front-end Pods should not care which back-end Pod it talks to, so long as it can talk to at least one of them. This idea of a Service allows for the front-end and back-end to be decoupled, but still provide the required functionality/communication.

4. Fun with kubectl

OK, lets now do something useful with kubectl. In this example, my master node has an IP address of 10.27.44.35, so all of the following commands need to point to the master.

First off, let’s get the version of K8S. This can be gleaned from the major and minor numbers shown below.

C:\Users\chogan>kubectl -s 10.27.44.35:8080 version
Client Version: version.Info{Major:"1", Minor:"1", GitVersion:"v1.1.3+$Format:%h$", \
GitCommit:"$Format:%H$", GitTreeState:"not a git tree"}Server Version: \
version.Info{Major:"1", Minor:"0", GitVersion:"v1.0.1", \
GitCommit:"6a5c06e3d1eb27a6310a09270e4a5fb1afa93e74", GitTreeState:"clean"}

Lets now look at the nodes (worker machines). We can see two nodes at the moment, a master and a slave. Nodes in this case correspond to the virtual machines.

C:\Users\chogan>kubectl -s 10.27.44.35:8080 get nodes
NAME                                          LABELS                                 \
                              STATUS    AGE
master-459e501f-1d17-4a89-b408-a78902fb6db2   kubernetes.io/hostname=master-459e501f-\
1d17-4a89-b408-a78902fb6db2   Ready     21m
slave-6cf96592-8a00-4365-a6c3-596e1f7706ea    kubernetes.io/hostname=slave-6cf96592-\
8a00-4365-a6c3-596e1f7706ea   Ready     20m

Let’s deploy our first container using kubectl. In this case, I am going to deploy a container with an image of NGINX can give it a name of nginx-app (nginx – engine-X – is essentially a web server). The DOMAIN is the name of my cluster. Note that this is now a replication controller, or “rc”.

C:\Users\chogan>kubectl -s 10.27.44.35:8080 run --image nginx nginx-app \
--port=80 --env="DOMAIN=Kube"
replicationcontroller "nginx-app" created

Next, I am going to create a service using the replication controller (rc) nginx-app and call it nginx-http.

C:\Users\chogan>kubectl -s 10.27.44.35:8080 expose rc nginx-app --port 80 \
--name=nginx-http
service "nginx-http" exposed

Now if I examine my pods, I can see the nginx-app pod, called nginx-app-qauw0:

C:\Users\chogan>kubectl -s 10.27.44.35:8080 get pods
NAME                                                    READY     STATUS    RESTARTS\
   AGE
k8s-master-master-459e501f-1d17-4a89-b408-a78902fb6db2  3/3       Running   0       \
   32m
nginx-app-qauw0                                         1/1       Running   0       \
   9m

And if I get additional detail about the replication controller, nginx-app, I can see that there is only a single replica running, meaning that the pod is not highly available. We will address that shortly.


C:\Users\chogan>kubectl -s 10.27.44.35:8080 get rc nginx-app
CONTROLLER   CONTAINER(S)   IMAGE(S)   SELECTOR        REPLICAS   AGE
nginx-app    nginx-app      nginx      run=nginx-app   1          18m

Additional details about the replication controller can be got using the following command:

C:\Users\chogan>kubectl -s 10.27.44.35:8080 describe rc nginx-app
Name:           nginx-app
Namespace:      default
Image(s):       nginx
Selector:       run=nginx-app
Labels:         run=nginx-app
Replicas:       1 current / 1 desired
Pods Status:    1 Running / 0 Waiting / 0 Succeeded / 0 Failed
No volumes.
Events:
  FirstSeen     LastSeen        Count   From                            SubobjectPath \
Reason          Message
  ─────────     ────────        ─────   ────                            ───────────── \
──────          ───────
  29m           29m             1       {replication-controller }                     \
successfulCreate        Created pod: nginx-app-qauw0

And if you wish to see which replication controllers and services are running, this information is also available:

C:\Users\chogan>kubectl -s 10.27.44.35:8080 get rc,services
CONTROLLER   CONTAINER(S)   IMAGE(S)   SELECTOR        REPLICAS   AGE
nginx-app    nginx-app      nginx      run=nginx-app   1          52m
NAME         CLUSTER_IP   EXTERNAL_IP   PORT(S)   SELECTOR        AGE
kubernetes   10.0.0.1     <none>        443/TCP   <none>          1h
nginx-http   10.0.0.246   <none>        80/TCP    run=nginx-app   46m

Administrators can also look at the services in further detail. Here you can see not only the Cluster IP for the services, but also the IP address endpoints on the container network.

C:\Users\chogan>kubectl -s 10.27.44.35:8080 describe service nginx-http
Name:                   nginx-http
Namespace:              default
Labels:                 run=nginx-app
Selector:               run=nginx-app
Type:                   ClusterIP
IP:                     10.0.0.246
Port:                   <unnamed>       80/TCP
Endpoints:              10.2.9.2:80
Session Affinity:       None
No events.

Let’s now look at how one can scale out the replica controllers as well as the worker machines (nodes/VMs).

5. Scaling out – replica controllers and worker machines

Let’s start by scaling out the number of replica controllers of the nginx-app to 2. We can use the following command.

C:\Users\chogan>kubectl -s 10.27.44.35:8080 scale --replicas=2 rc nginx-app
replicationcontroller "nginx-app" scaled

Let’s run a command that we ran previously and examine the number of replicas. Now we can see that it has increased to 2, meaning the nginx-app pod is now more highly available.

C:\Users\chogan>kubectl -s 10.27.44.35:8080 get rc nginx-app
CONTROLLER   CONTAINER(S)   IMAGE(S)   SELECTOR        REPLICAS   AGE
nginx-app    nginx-app      nginx      run=nginx-app   2          1h

C:\Users\chogan>

And using the “describe” option, we can see that the number of pods for this application has increased to 2:

C:\Users\chogan>kubectl -s 10.27.44.35:8080 describe rc nginx-app
Name:           nginx-app
Namespace:      default
Image(s):       nginx
Selector:       run=nginx-app
Labels:         run=nginx-app
Replicas:       2 current / 2 desired
Pods Status:    2 Running / 0 Waiting / 0 Succeeded / 0 Failed
No volumes.
Events:
  FirstSeen     LastSeen        Count   From                            SubobjectPath\
   Reason          Message
  ─────────     ────────        ─────   ────                            ─────────────\
   ──────          ───────
  14m           14m             1       {replication-controller }                    \
   successfulCreate        Created pod: nginx-app-tyw0q

Let’s use the Photon Controller CLI to add some additional worker machines, in other words, new VMs for running more containers:

C:\Users\chogan>photon cluster resize 54578909-6317-4d7d-ab80-005a2f0e9f60 3
Using target 'http://10.27.44.34:28080'

Resizing cluster 54578909-6317-4d7d-ab80-005a2f0e9f60 to slave count 3
Are you sure [y/n]? y
RESIZE_CLUSTER completed for '' entity
Note: A background task is running to gradually resize the cluster to its target 
capacity.
You may continue to use the cluster. You can run 'cluster show '
to see the state of the cluster. If the resize operation is still in progress, 
the cluster state
will show as RESIZING. Once the cluster is resized, the cluster state will show 
as READY.

C:\Users\chogan>

This may take some time as images for the slave may have to be copied to a new ESXi CLOUD host, if a previous image doesn’t exist, and the -i EAGER option was not used when the image was initially created. During the creation of new slaves, the cluster status changes to RESIZING:

C:\Users\chogan>photon cluster show 54578909-6317-4d7d-ab80-005a2f0e9f60
Using target 'http://10.27.44.34:28080'
Cluster ID:             54578909-6317-4d7d-ab80-005a2f0e9f60
  Name:                 Kube
  State:                RESIZING
  Type:                 KUBERNETES
  Slave count:          3
  Extended Properties:  map[netmask:255.255.240.0 dns:10.16.142.110 \
etcd_ips:10.27.44.36 master_ip:10.27.44.35 gateway:10.27.47.254 \
container_network:10.2.0.0/16]

VM ID                                 VM Name                                      \
VM IP
074d87d6-e259-42e2-a68a-ddb4f4e88afe  master-459e501f-1d17-4a89-b408-a78902fb6db2  \
10.27.44.35
b3093ee6-b02f-4005-b7ab-bed9ef439d0d  etcd-13cf02af-a5f7-47b6-826c-937d018312ad    \
10.27.44.36

When the list of VMs is queried, you may notice that some have “started” and some are in a state of “creating”. In this example, the creating status was due to the image being uploaded to a new ESXi Cloud host where the slave VM/node was being deployed.

C:\Users\chogan>photon cluster list_vms 54578909-6317-4d7d-ab80-005a2f0e9f60
Using target 'http://10.27.44.34:28080'
ID                                    Name                                         State
074d87d6-e259-42e2-a68a-ddb4f4e88afe  master-459e501f-1d17-4a89-b408-a78902fb6db2  STARTED
1130116f-a8e8-4a8d-aedf-7d928bcd0637  slave-63982978-b4e2-418f-b2a5-480864758904   CREATING
18eca5ba-da65-4f8a-b2d4-f2d426b83317  slave-6cf96592-8a00-4365-a6c3-596e1f7706ea   STARTED
b3093ee6-b02f-4005-b7ab-bed9ef439d0d  etcd-13cf02af-a5f7-47b6-826c-937d018312ad    STARTED
d6c539a0-8e33-4364-a309-31d8b57f63ab  slave-12ce90c4-1652-4240-b8c2-57805d734c85   STARTED
Total: 5
STARTED: 4
CREATING: 1

C:\Users\chogan>

If we flip back to the kubectl command, we can now see that there is an additional slave node. The one in a state of “creating” above has not been added yet.

C:\Users\chogan>kubectl -s 10.27.44.35:8080 get nodes
NAME                                          LABELS                                \
                               STATUS    AGE
master-459e501f-1d17-4a89-b408-a78902fb6db2   kubernetes.io/hostname=master-459e501f\
-1d17-4a89-b408-a78902fb6db2   Ready     1h
slave-12ce90c4-1652-4240-b8c2-57805d734c85    kubernetes.io/hostname=slave-12ce90c4\
-1652-4240-b8c2-57805d734c85    Ready     5m
slave-6cf96592-8a00-4365-a6c3-596e1f7706ea    kubernetes.io/hostname=slave-6cf96592\
-8a00-4365-a6c3-596e1f7706ea    Ready     1h

C:\Users\chogan>

Eventually, when the image uploads and the new slave is created, we should see all of the slaves in a “started” state, and kubectl should show us the new slave also:

C:\Users\chogan>photon cluster list_vms 54578909-6317-4d7d-ab80-005a2f0e9f60
Using target 'http://10.27.44.34:28080'
ID                                    Name                                         State
074d87d6-e259-42e2-a68a-ddb4f4e88afe  master-459e501f-1d17-4a89-b408-a78902fb6db2  STARTED
18eca5ba-da65-4f8a-b2d4-f2d426b83317  slave-6cf96592-8a00-4365-a6c3-596e1f7706ea   STARTED
9f197d88-2e50-40a0-9a31-4d628ca526bb  slave-3590320e-d93c-4b15-82e3-90df740a6a68   STARTED
b3093ee6-b02f-4005-b7ab-bed9ef439d0d  etcd-13cf02af-a5f7-47b6-826c-937d018312ad    STARTED
d6c539a0-8e33-4364-a309-31d8b57f63ab  slave-12ce90c4-1652-4240-b8c2-57805d734c85   STARTED
Total: 5
STARTED: 5

C:\Users\chogan>kubectl -s 10.27.44.35:8080 get nodes
NAME                                          LABELS                               \
                                STATUS    AGE
master-459e501f-1d17-4a89-b408-a78902fb6db2   kubernetes.io/hostname=master-459e501f\
-1d17-4a89-b408-a78902fb6db2   Ready     2h
slave-12ce90c4-1652-4240-b8c2-57805d734c85    kubernetes.io/hostname=slave-12ce90c4\
-1652-4240-b8c2-57805d734c85    Ready     26m
slave-3590320e-d93c-4b15-82e3-90df740a6a68    kubernetes.io/hostname=slave-3590320e\
-d93c-4b15-82e3-90df740a6a68    Ready     1m
slave-6cf96592-8a00-4365-a6c3-596e1f7706ea    kubernetes.io/hostname=slave-6cf96592\
-8a00-4365-a6c3-596e1f7706ea    Ready     2h

C:\Users\chogan>

6. Removing the cluster

In this last step, I will tear down the K8S configuration, and also remove the cluster using the Photon CLI command.

First, I will delete the nginx-http service:

C:\Users\chogan>kubectl -s 10.27.44.35:8080 delete service nginx-http
service "nginx-http" deleted

Next, I will remove the nginx-app replication controller:

C:\Users\chogan>kubectl -s 10.27.44.35:8080 delete rc nginx-app
replicationcontroller "nginx-app" deleted

If I check the Pods, I will now only see the master Pod. The nginx-app pod is removed:

C:\Users\chogan>kubectl -s 10.27.44.35:8080 get pods
NAME                                                     READY   STATUS   RESTARTS  AGE
k8s-master-master-459e501f-1d17-4a89-b408-a78902fb6db2   3/3     Running  0         2h

Now I can use the Photon CLI command to delete the cluster:

C:\Users\chogan>photon cluster delete 54578909-6317-4d7d-ab80-005a2f0e9f60
Using target 'http://10.27.44.34:28080'

Deleting cluster 54578909-6317-4d7d-ab80-005a2f0e9f60
Are you sure [y/n]? y
DELETE_CLUSTER completed for '' entity

C:\Users\chogan>

Once again, I hope this shows you how Photon Controller allows you to use ESXi resources for container frameworks such as Kubernetes. In this case K8S “worker machines” are running as VMs. Using the Photon CLI, you can very quickly deploy Photon Controller, and then create a K8S cluster on top. And then, it is just like using K8S as you normally would for container deployment and management. The set of “kubectl” examples will hopefully give you some idea of what you can do, and allow you to go on and explore the capabilities in further detail.