Deploy Kubernetes on Photon Platform 1.2 and VSAN

To complete my series of posts on Photon Platform version 1.2, my next step is to deploy Kubernetes (version 1.6) and use my vSAN datastore as the storage destination. The previous posts covered the new Photon Platform v1.2 deployment model, and I also covered how to setup vSAN and make the datastore available to the cloud hosts in Photon Platform v1.2. This final step will use the photon controller CLI (mostly) for creating the tenant, project,  image, and all the other steps that are required for deploying K8S on vSAN via PPv1.2. I’m very much going to include a warts-n-all approach to this deployment, as there was a lot of trial and error. A few things have changes with the new v1.2, especially with the use of quotas, which replaces the old resource ticket. The nice thing about quotas is that they can be adjusted on the fly, but they take a bit of getting used to.

What you need:

  1. Photon Platform v1.2 should already be deployed.
  2. You need 3 static ip addresses for Kubernetes (K8S) VMs, a master ip, a load-balancer ip and an ip address for etcd.
  3. The network on which K8S VMs is deployed needs DHCP for the worker VMs.
  4. You need photon controller CLI installed on your desktop/laptop to run photon CLI commands.

Why not use the UI?

Yes – you can certainly do this, and I showed how to deploy Kubernetes-As-A-Service with Photon Platform version 1.1. Unfortunately there seems to be an issue with the UI in Photon Platform 1.2, where the DNS, gateway and netmask are lost on the Summary tab, so does not complete. I fed this back to the team, and I’m sure it’ll be addressed soon, so in the meantime, the CLI is the way forward (for most of what I need to do).

Step 1 – Get photon controller CLI and K8S image

You can get these from the usual place on GitHub. Deploy it to your desktop, as we will use this CLI to complete the deployment of K8S.

Step 2 – Set the target and login

Using the photon controller CLI (I am using the Mac version), set the target to your Photon Platform Load Balancer, and login. Use https/port 443 as shown here:

Cormacs-MacBook-Pro-8:bin cormachogan$ photon target set -c https://10.27.51.68:443

API target set to 'https://10.27.51.68:443'

Cormacs-MacBook-Pro-8:bin cormachogan$ photon target login
User name (username@tenant): administrator@rainpole.local
Password:
Login successful

Step 3 – Upload the K8S image to PPV1.2

The assumption is that the K8S OVA has been downloaded locally from step 1. I’ll now push it up to PP using the following command, and check it afterwards.
Cormacs-MacBook-Pro-8:bin cormachogan$ photon image create PP1.2/kubernetes-1.6.0-pc-1.2.1-77a6d82.ova \
-n kube1 -i EAGER
Project ID (ENTER to create infrastructure image):
CREATE_IMAGE completed for 'image' entity 91a1bed4-1b54-4c9e-82cd-392ae3f1ae9b
Cormacs-MacBook-Pro-8:bin cormachogan$ photon image list
ID                                    Name   State  Size(Byte)   Replication_type  ReplicationProgress  SeedingProgress  Scope
91a1bed4-1b54-4c9e-82cd-392ae3f1ae9b  kube1  READY  41943040096  EAGER             50%                  50%              infrastructure
Total: 1

Step 4 – Tag the image for K8S

I tried to do this via the photon controller CLI, but it wouldn’t work for me.

Cormacs-MacBook-Pro-8:bin cormachogan$ photon deployment enable-cluster-type
2017/05/25 13:45:19 Error: photon: { HTTP status: '404', code: 'NotFound', message: '', data: 'map[]’ }

I’ve also provided feedback to the team on this. It looks like a known issue which will be addressed in the next update of photon controller CLI. Eventually, to unblock myself, I just logged onto the Photon Platform UI, and marked the image for Kubernetes that way. This is pretty easy to do; just select the image, then actions followed by “Use as Kubernetes image”.

Step 5 – Create a network for the VMs that will run the K8S scheduler/orchestration framework

This next step is needed so that the VMs that run the various K8S containers (master, etcd, local-balancer and workers) can all communicate. We’re basically creating a network for the VMs to use. I simply leveraged the default VM network in this example (which also has DHCP, as some of the K8S worker VMs pick up IPs from DHCP). This network is specified when we are ready to deploy the K8S ‘service’.

Cormacs-MacBook-Pro-8:bin cormachogan$ photon subnet create --name vm-network --portgroups 'VM Network'
Subnet Description: k8s-network
CREATE_NETWORK completed for 'network' entity 94660bdd-d1e1-45b5-8e80-8d737915d751                 

Cormacs-MacBook-Pro-8:bin cormachogan$ photon subnet set-default 94660bdd-d1e1-45b5-8e80-8d737915d751
Are you sure [y/n]? y
SET_DEFAULT_NETWORK completed for 'network' entity 94660bdd-d1e1-45b5-8e80-8d737915d751            

Cormacs-MacBook-Pro-8:bin cormachogan$ photon subnet list
ID                                    Name        Kind    Description  PrivateIpCidr  ReservedIps  State  IsDefault  PortGroups
94660bdd-d1e1-45b5-8e80-8d737915d751  vm-network  subnet  k8s-network                 map[]        READY  true       [VM Network]
Cormacs-MacBook-Pro-8:bin cormachogan$ 

Step 6 – Create a “flavour” for the K8S VMs

This step simply defines the resources that go to make up the VMs that are deployed to run the K8S containers. I’ve made it quite small, 1 CPU and 2GB memory. This will also be used at the command line when deploying K8S later on.

Cormacs-MacBook-Pro-8:bin cormachogan$ photon flavor create --name \
cluster-small -k vm --cost 'vm 1 COUNT, vm.cpu 1 COUNT, vm.memory 2 GB'
Creating flavor: 'cluster-small', Kind: 'vm'

Please make sure limits below are correct: 
1: vm, 1, COUNT
2: vm.cpu, 1, COUNT
3: vm.memory, 2, GB
Are you sure [y/n]? y
CREATE_FLAVOR completed for 'vm' entity 898322d6-2d9d-4a84-9f41-f0f9adf76cb1                       
Cormacs-MacBook-Pro-8:bin cormachogan$

Step 7 – Tenant, Project and fun with Quotas

In this step, I create the tenant, the project and then select some quotas (resources).  I said I would show the “warts and all” approach, so here you will see the various errors I get when I don’t get the quota settings quite right. When you create the tenant and project, you can use the –limits option to specify a quota, or you can use the ‘quota update‘ to modify the resources afterwards. I will show an update example later. Let’s create the tenant with some quota. I will set the tenant to have 100 VMs, 1TB memory and 500 CPUs. I will then create a project to consume part of those VM CPU and Memory resources. There could be multiple projects in a tenant, but in this setup, there is only one. I will then set the tenant and project context to point to these.

Cormacs-MacBook-Pro-8:bin cormachogan$ photon tenant quota set plato --limits\
 'vm.count 100 COUNT, vm.memory 1000 GB, vm.cpu 500 COUNT'

Tenant name: plato
Please make sure limits below are correct:
1: vm.count, 100, COUNT
2: vm.memory, 1000, GB
3: vm.cpu, 500, COUNT
Are you sure [y/n]? y
RESET_QUOTA completed for 'tenant' entity ebdd8cd4-515d-4f71-8ce1-a7e9cb318dba 

Cormacs-MacBook-Pro-8:bin cormachogan$ photon tenant quota show plato
    Limits:
      vm.count   100   COUNT
      vm.cpu     500   COUNT
      vm.memory  1000  GB
    Usage:
      vm.count   0   COUNT
      vm.cpu     0   COUNT
      vm.memory  0   GB

Cormacs-MacBook-Pro-8:bin cormachogan$ photon project create plato-prjt \
--tenant plato --limits 'vm.memory 100 GB, vm.cpu 50 COUNT, vm.count 10 COUNT

Tenant name: plato
Creating project name: plato-prjt

Please make sure limits below are correct:
1: vm.memory, 100, GB
2: vm.cpu, 50, COUNT
3: vm.count, 10, COUNT
Are you sure [y/n]? y
CREATE_PROJECT completed for 'project' entity 7592c6f4-07f5-450f-83d7-267af1acd48e                 
Cormacs-MacBook-Pro-8:bin cormachogan$ 

Cormacs-MacBook-Pro-8:bin cormachogan$ photon tenant quota show plato
    Limits:
      vm.count   100   COUNT
      vm.cpu     500   COUNT
      vm.memory  1000  GB
    Usage:
      vm.cpu     50   COUNT
      vm.memory  100  GB
      vm.count   10   COUNT
Cormacs-MacBook-Pro-8:bin cormachogan$ 

Cormacs-MacBook-Pro-8:bin cormachogan$ photon tenant set plato
Tenant set to ‘plato’

Cormacs-MacBook-Pro-8:bin cormachogan$ photon project set plato-prjt
Project set to 'plato-prjt'
Cormacs-MacBook-Pro-8:bin cormachogan$ 

Now, what I did not include here is anything related to disk resources. Let’s do that next. I’ll add persistent disks and ephemeral disks to the tenant and its project. See if you can spot my not so deliberate mistake:

Cormacs-MacBook-Pro-8:bin cormachogan$ photon tenant quota update plato \
--limits 'persistent-disk 100 COUNT, persistent-disk.capacity 200 GB'

Tenant name: plato
Please make sure limits below are correct:
1: persistent-disk, 100, COUNT
2: persistent-disk.capacity, 200, GB
Are you sure [y/n]? y
UPDATE_QUOTA completed for 'tenant' entity ebdd8cd4-515d-4f71-8ce1-a7e9cb318dba                    

Cormacs-MacBook-Pro-8:bin cormachogan$ photon tenant quota update plato \
--limits 'ephereral-disk 100 COUNT, ephereral-disk.capacity 200 GB'

Tenant name: plato
Please make sure limits below are correct:
1: ephereral-disk, 100, COUNT
2: ephereral-disk.capacity, 200, GB
Are you sure [y/n]? y
UPDATE_QUOTA completed for 'tenant' entity ebdd8cd4-515d-4f71-8ce1-a7e9cb318dba                    

Cormacs-MacBook-Pro-8:bin cormachogan$ photon project quota update \
--limits 'persistent-disk 100 COUNT, persistent-disk.capacity 200 GB, \
ephereral-disk 100 COUNT, ephereral-disk.capacity 200 GB' 7592c6f4-07f5-450f-83d7-267af1acd48e

Project Id: 7592c6f4-07f5-450f-83d7-267af1acd48e
Please make sure limits below are correct:
1: persistent-disk, 100, COUNT
2: persistent-disk.capacity, 200, GB
3: ephereral-disk, 100, COUNT
4: ephereral-disk.capacity, 200, GB
Are you sure [y/n]? y
UPDATE_QUOTA completed for 'project' entity 7592c6f4-07f5-450f-83d7-267af1acd48e                   
Cormacs-MacBook-Pro-8:bin cormachogan$

Step 8 –  Let’s deploy some Kubernetes

OK – I thought this would be more interesting if I tried some trial and error stuff here. I didn’t spend too much time worrying about whether I got the quotas right, and so on. I simply decided to try various options and see how it erred out. Feel free to skip down to Take #5 if you’re not interested in this approach.

Take #1: Most of this command be straight-forward I think. The only thing that might be confusing is the -d “vsan-disk”. I’ll come back to that shortly. However it looks like we’ve failed due to a lack of resources. Note that the command line is picking up the default tenant, project and network. We are specifing the vm_flavor created earlier though. Most of the rest of the command is taken up with the ip addresses of the K8S master, etcd and load-balancer.

Cormacs-MacBook-Pro-8:bin cormachogan$ photon service create -n kube-socrates -k KUBERNETES --master-ip 10.27.51.208 \
--etcd1 10.27.51.209 --load-balancer-ip 10.27.51.210 --container-network 10.2.0.0/16 --dns 10.27.51.35 \
--gateway 10.27.51.254 --netmask 255.255.255.0 -c 2 --vm_flavor cluster-small --ssh-key ~/.ssh/id_rsa.pub -d vsan-disk
Kubernetes master 2 static IP address (leave blank for none):
etcd server 2 static IP address (leave blank for none):

Creating service: kube-socrates (KUBERNETES)
  Disk flavor: vsan-disk
  Worker count: 5

Are you sure [y/n]? y
2017/05/25 14:37:26 Error: photon: Task 'd75ca8b9-8a09-48bf-8e94-4718702c9a73' is in error state: \
{@step=={"sequence"=>"1","state"=>"ERROR","errors"=>[photon: { HTTP status: '0', code: 'InternalError', \
message: 'Failed to rollout KubernetesEtcd. Error: MultiException[java.lang.IllegalStateException: \
VmProvisionTaskService failed with error com.vmware.photon.controller.api.frontend.exceptions.external.QuotaException: \
Not enough quota: Current Limit: vm, 0.0, COUNT, desiredUsage vm, 1.0, COUNTQuotaException{limit.key=vm, limit.value=0.0, \
limit.unit=COUNT, usage.key=vm, usage.value=0.0, usage.unit=COUNT, newUsage.key=vm, newUsage.value=1.0, newUsage.unit=COUNT}. \
/photon/servicesmanager/vm-provision-tasks/48ef339d5505951ddff41]', data: 'map[]' }],"warnings"=>[],"operation"=>"\
CREATE_KUBERNETES_SERVICE_SETUP_ETCD","startedTime"=>"1495719439272","queuedTime"=>"1495719439232","endTime"=>"1495719444273","options"=>map[]}}
API Errors: [photon: { HTTP status: '0', code: 'InternalError', message: 'Failed to rollout KubernetesEtcd. Error: \
MultiException[java.lang.IllegalStateException: VmProvisionTaskService failed with error com.vmware.photon.controller.\
api.frontend.exceptions.external.QuotaException: Not enough quota: Current Limit: vm, 0.0, COUNT, desiredUsage vm, 1.0, \
COUNTQuotaException{limit.key=vm, limit.value=0.0, limit.unit=COUNT, usage.key=vm, usage.value=0.0, usage.unit=COUNT, \
newUsage.key=vm, newUsage.value=1.0, newUsage.unit=COUNT}. /photon/servicesmanager/vm-provision-tasks/48ef339d5505951ddff41]', data: 'map[]' }]
Cormacs-MacBook-Pro-8:bin cormachogan>

I’ve hit my first error. The description is pretty good though. The issue here is that I never defined a quota for VM. Let’s address that:

Cormacs-MacBook-Pro-8:bin cormachogan$ photon tenant quota update plato \
--limits 'vm 100 COUNT'

Tenant name: plato
Please make sure limits below are correct:
1: vm, 100, COUNT
Are you sure [y/n]? y
UPDATE_QUOTA completed for 'tenant' entity ebdd8cd4-515d-4f71-8ce1-a7e9cb318dba                    
Cormacs-MacBook-Pro-8:bin cormachogan$ photon tenant quota show plato
    Limits:
      ephereral-disk.capacity   200        GB
      persistent-disk           100        COUNT
      ephemeral-disk            200        COUNT
      persistent-disk.capacity  200        GB
      ephereral-disk            100        COUNT
      vm.count                  100        COUNT
      vm.cpu                    500        COUNT
      vm.memory                 1000       GB
      ephemeral-disk.capacity   1.024e+06  MB
      vm                        100        COUNT
    Usage:
      ephereral-disk.capacity   200  GB
      persistent-disk           100  COUNT
      vm                        0    COUNT
      ephemeral-disk            0    COUNT
      persistent-disk.capacity  200  GB
      ephereral-disk            100  COUNT
      vm.count                  10   COUNT
      vm.cpu                    50   COUNT
      vm.memory                 100  GB
      ephemeral-disk.capacity   0    MB

Cormacs-MacBook-Pro-8:bin cormachogan$ photon project quota update \
--limits 'vm 100 COUNT' 7592c6f4-07f5-450f-83d7-267af1acd48e

Project Id: 7592c6f4-07f5-450f-83d7-267af1acd48e
Please make sure limits below are correct:
1: vm, 100, COUNT
Are you sure [y/n]? y
UPDATE_QUOTA completed for 'project' entity 7592c6f4-07f5-450f-83d7-267af1acd48e

Take #2: With that issue addressed, let’s try once more.

Cormacs-MacBook-Pro-8:bin cormachogan$ photon service create -n kube-socrates -k KUBERNETES --master-ip 10.27.51.208 \
--etcd1 10.27.51.209 --load-balancer-ip 10.27.51.210 --container-network 10.2.0.0/16 --dns 10.27.51.252 \
--gateway 10.27.51.254 --netmask 255.255.255.0 -c 5 --vm_flavor cluster-small --ssh-key ~/.ssh/id_rsa.pub -d vsan-disk
Kubernetes master 2 static IP address (leave blank for none):
etcd server 2 static IP address (leave blank for none):

Creating service: kube-socrates (KUBERNETES)
  Disk flavor: vsan-disk
  Worker count: 5

Are you sure [y/n]? y
2017/05/25 15:42:24 Error: photon: Task 'f79cfd7f-03f0-4112-b156-15939891ddcb' is in error state: \
{@step=={"sequence"=>"1","state"=>"ERROR","errors"=>[photon: { HTTP status: '0', code: 'InternalError', \
message: 'Failed to rollout KubernetesEtcd. Error: MultiException[java.lang.IllegalStateException: \
VmProvisionTaskService failed with error com.vmware.photon.controller.api.frontend.exceptions.external.\
FlavorNotFoundException: Flavor vsan-disk is not found for kind ephemeral-disk. /photon/servicesmanager/\
vm-provision-tasks/48ef339d5505a3a2cf5b0]', data: 'map[]' }],"warnings"=>[],"operation"=>"\
CREATE_KUBERNETES_SERVICE_SETUP_ETCD","startedTime"=>"1495723336805","queuedTime"=>"1495723336795",\
"endTime"=>"1495723341806","options"=>map[]}}
API Errors: [photon: { HTTP status: '0', code: 'InternalError', message: 'Failed to rollout KubernetesEtcd. \
Error: MultiException[java.lang.IllegalStateException: VmProvisionTaskService \
failed with error com.vmware.photon.controller.api.frontend.exceptions.external.FlavorNotFoundException: \
Flavor vsan-disk is not found for kind ephemeral-disk. /photon/servicesmanager/vm-provision-tasks/48ef339d5505a3a2cf5b0]', \
data: 'map[]' }]

Ah – so this is a problem with the vsan-disk option. I actually need to make a flavor for this so that it places the VMs on vSAN storage.  Here is how we do that:

Cormacs-MacBook-Pro-8:bin cormachogan$ photon -n flavor create --name "vsan-disk" \
--kind "ephemeral-disk" --cost "storage.VSAN 1.0 COUNT"
167951af-8bfa-4746-b58f-810157dc4bf8

Cormacs-MacBook-Pro-8:bin cormachogan$ photon -n flavor list
167951af-8bfa-4746-b58f-810157dc4bf8 vsan-disk ephemeral-disk storage.VSAN:1:COUNT
 898322d6-2d9d-4a84-9f41-f0f9adf76cb1 cluster-small vm vm:1:COUNT,vm.cpu:1:COUNT,vm.memory:2:GB
service-flavor-1 service-master-vm vm vm.count:1:COUNT,vm.cpu:4:COUNT,vm.memory:8:GB
service-flavor-2 service-other-vm vm vm.count:1:COUNT,vm.cpu:1:COUNT,vm.memory:4:GB
service-flavor-3 service-vm-disk ephemeral-disk ephemeral-disk:1:COUNT
service-flavor-4 service-generic-persistent-disk persistent-disk persistent-disk:1:COUNT
service-flavor-5 service-local-vmfs-persistent-disk persistent-disk storage.LOCAL_VMFS:1:COUNT
service-flavor-6 service-shared-vmfs-persistent-disk persistent-disk storage.SHARED_VMFS:1:COUNT
service-flavor-7 service-vsan-persistent-disk persistent-disk storage.VSAN:1:COUNT
service-flavor-8 service-nfs-persistent-disk persistent-disk storage.NFS:1:COUNT
Cormacs-MacBook-Pro-8:bin cormachogan$

Take #3: Let’s try again.

Cormacs-MacBook-Pro-8:bin cormachogan$ photon service create -n kube-socrates -k KUBERNETES --master-ip 10.27.51.208 \
--etcd1 10.27.51.209 --load-balancer-ip 10.27.51.210 --container-network 10.2.0.0/16 --dns 10.27.51.252 \
--gateway 10.27.51.254 --netmask 255.255.255.0 -c 5 --vm_flavor cluster-small --ssh-key ~/.ssh/id_rsa.pub -d vsan-disk

Kubernetes master 2 static IP address (leave blank for none):

 etcd server 2 static IP address (leave blank for none):

Creating service: kube-socrates (KUBERNETES)
 Disk flavor: vsan-disk
 Worker count: 5

Are you sure [y/n]? y
 2017/05/25 15:47:01 Error: photon: Task 'e0eff9c7-9f41-4d1d-b92d-3b5e854da717' is in error state: {@step=={"sequence"=>"1",\
"state"=>"ERROR","errors"=>[photon: { HTTP status: '0', code: 'InternalError', message: 'Failed to rollout KubernetesEtcd. \
Error: MultiException[java.lang.IllegalStateException: VmProvisionTaskService failed with error com.vmware.photon.controller.\
api.frontend.exceptions.external.QuotaException: Not enough quota: Current Limit: storage.VSAN, 0.0, COUNT, desiredUsage storage.VSAN, 1.0, \
COUNTQuotaException{limit.key=storage.VSAN, limit.value=0.0, limit.unit=COUNT, usage.key=storage.VSAN, usage.value=0.0, usage.unit=COUNT, \
newUsage.key=storage.VSAN, newUsage.value=1.0, newUsage.unit=COUNT}. /photon/servicesmanager/vm-provision-tasks/48ef339d5505a4aad7272]', \
data: 'map[]' }],"warnings"=>[],"operation"=>"CREATE_KUBERNETES_SERVICE_SETUP_ETCD","startedTime"=>"1495723613645","queuedTime"=>"1495723613626",\
"endTime"=>"1495723618649","options"=>map[]}}
 API Errors: [photon: { HTTP status: '0', code: 'InternalError', message: 'Failed to rollout KubernetesEtcd. Error: MultiException\
[java.lang.IllegalStateException: VmProvisionTaskService failed with error com.vmware.photon.controller.api.frontend.exceptions.external.\
QuotaException: Not enough quota: Current Limit: storage.VSAN, 0.0, COUNT, desiredUsage storage.VSAN, 1.0, COUNTQuotaException\
{limit.key=storage.VSAN, limit.value=0.0, limit.unit=COUNT, usage.key=storage.VSAN, usage.value=0.0, usage.unit=COUNT, newUsage.key=storage.VSAN, \
newUsage.value=1.0, newUsage.unit=COUNT}. /photon/servicesmanager/vm-provision-tasks/48ef339d5505a4aad7272]', data: 'map[]' }]

Another quota issue, this time against the newly created vsan-disk. Let’s just bump up the count (and capacity while I am at it).

Cormacs-MacBook-Pro-8:bin cormachogan$ photon project quota update --limits 'storage.VSAN 100 COUNT, \
storage.VSAN.capacity 1000 GB' 7592c6f4-07f5-450f-83d7-267af1acd48e

Project Id: 7592c6f4-07f5-450f-83d7-267af1acd48e
Please make sure limits below are correct:
1: storage.VSAN, 100, COUNT
2: storage.VSAN.capacity, 1000, GB
Are you sure [y/n]? y
UPDATE_QUOTA completed for 'project' entity 7592c6f4-07f5-450f-83d7-267af1acd48e                   
Cormacs-MacBook-Pro-8:bin cormachogan$

Take #4. Once more into the breach:

Cormacs-MacBook-Pro-8:bin cormachogan$ photon service create -n kube-socrates -k KUBERNETES --master-ip 10.27.51.208 \
 --etcd1 10.27.51.209 --load-balancer-ip 10.27.51.210 --container-network 10.2.0.0/16 --dns 10.27.51.252 \
 --gateway 10.27.51.254 --netmask 255.255.255.0 -c 5 --vm_flavor cluster-small --ssh-key ~/.ssh/id_rsa.pub -d vsan-disk
 Kubernetes master 2 static IP address (leave blank for none):
 etcd server 2 static IP address (leave blank for none):

Creating service: kube-socrates (KUBERNETES)
   Disk flavor: vsan-disk
   Worker count: 5

Are you sure [y/n]? y
 2017/05/25 15:50:32 Error: photon: Task '1ac181b3-fc07-4313-8fbe-d1900d8de01c' is in error state: {@step=={"sequence"=>"1","state"=>\
 "ERROR","errors"=>[photon: { HTTP status: '0', code: 'InternalError', message: 'Failed to rollout KubernetesEtcd. Error: MultiException\
 [java.lang.IllegalStateException: VmProvisionTaskService failed with error com.vmware.photon.controller.\api.frontend.exceptions.external\
 .QuotaException: Not enough quota: Current Limit: ephemeral-disk.capacity, 0.0, GB, desiredUsage ephemeral-disk.capacity, 39.0, GB\
 QuotaException{limit.key=ephemeral-disk.capacity, limit.value=0.0, limit.unit=GB, usage.key=ephemeral-disk.capacity, usage.value=0.0, \
 usage.unit=GB, newUsage.key=ephemeral-disk.capacity, newUsage.value=39.0, newUsage.unit=GB}. /photon/servicesmanager/vm-provision-tasks/\
 48ef339d5505a573b312a]', data: 'map[]' }],"warnings"=>[],"operation"=>"CREATE_KUBERNETES_SERVICE_SETUP_ETCD","startedTime"=>"1495723824262",\
 "queuedTime"=>"1495723824251","endTime"=>"1495723829263","options"=>map[]}}
 API Errors: [photon: { HTTP status: '0', code: 'InternalError', message: 'Failed to rollout KubernetesEtcd. Error: MultiException\
 [java.lang.IllegalStateException: VmProvisionTaskService failed with error com.vmware.photon.controller.api.frontend.exceptions.external.\
 QuotaException: Not enough quota: Current Limit: ephemeral-disk.capacity, 0.0, GB, desiredUsage ephemeral-disk.capacity, 39.0, GBQuotaException\
 {limit.key=ephemeral-disk.capacity, limit.value=0.0, limit.unit=GB, usage.key=ephemeral-disk.capacity, usage.value=0.0, usage.unit=GB, \
 newUsage.key=ephemeral-disk.capacity, newUsage.value=39.0, newUsage.unit=GB}. /photon/servicesmanager/vm-provision-tasks/48ef339d5505a573b312a]', data: 'map[]' }]

OK – this is going back to my deliberate mistake in step 7 – I typo’ed ephereral instead of ephemeral so I actually don’t have any ephemeral disk capacity. And my vSAN storage has been marked as kind “ephemeral”. Let’s now fix that one up:

Cormacs-MacBook-Pro-8:bin cormachogan$ photon tenant quota update plato --limits 'ephemeral-disk.capacity 200 GB'

Tenant name: plato
Please make sure limits below are correct:
1: ephemeral-disk.capacity, 200, GB
Are you sure [y/n]? y
UPDATE_QUOTA completed for 'tenant' entity ebdd8cd4-515d-4f71-8ce1-a7e9cb318dba                    


Cormacs-MacBook-Pro-8:bin cormachogan$ photon project quota update --limits 'ephemeral-disk.capacity 200 GB' \
7592c6f4-07f5-450f-83d7-267af1acd48e

Project Id: 7592c6f4-07f5-450f-83d7-267af1acd48e
Please make sure limits below are correct:
1: ephemeral-disk.capacity, 200, GB
Are you sure [y/n]? y
UPDATE_QUOTA completed for 'project' entity 7592c6f4-07f5-450f-83d7-267af1acd48e                   
Cormacs-MacBook-Pro-8:bin cormachogan$

Take #5 – Now it should work, right?

Cormacs-MacBook-Pro-8:.kube cormachogan$ photon service create -n kube-socrates -k KUBERNETES --master-ip 10.27.51.208 \
--etcd1 10.27.51.209 --load-balancer-ip 10.27.51.210 --container-network 10.2.0.0/16 --dns 10.27.51.252 \
--gateway 10.27.51.254 --netmask 255.255.255.0 -c 2 --vm_flavor cluster-small --ssh-key ~/.ssh/id_rsa.pub -d vsan-disk
Kubernetes master 2 static IP address (leave blank for none):
etcd server 2 static IP address (leave blank for none):

Creating service: kube-socrates (KUBERNETES)
  Disk flavor: vsan-disk
  Worker count: 2

Are you sure [y/n]? y
CREATE_SERVICE completed for 'service' entity 6be5ce48-81ff-4682-a5de-1066fb3fa316                 
Note: the service has been created with minimal resources. You can use the service now.
A background task is running to gradually expand the service to its target capacity.
You can run 'service show ' to see the state of the service.
Cormacs-MacBook-Pro-8:.kube cormachogan$

Success!

Right, let’s monitor the deployment while the Kubernetes workers are getting deployed. We should see it go from ‘Maintenance’ to “Ready’:

Cormacs-MacBook-Pro-8:.kube cormachogan$ photon service list
ID                                    Name           Type        State        Worker Count
6be5ce48-81ff-4682-a5de-1066fb3fa316  kube-socrates  KUBERNETES  MAINTENANCE  2

Total: 1
MAINTENANCE: 1

Cormacs-MacBook-Pro-8:.kube cormachogan$ photon service list
ID                                    Name           Type        State  Worker Count
6be5ce48-81ff-4682-a5de-1066fb3fa316  kube-socrates  KUBERNETES  READY  2

Total: 1
READY: 1

Step 9: Check out the Photon Platform Dashboard

The Photon Platform dashboard can now be used to look at the new cluster and the running VMs. Here are some screenshots from my setup. the first is the Tenant > Project > Cluster view where you can see the Kubernetes service, and the second is the Tenant > Project > VMs view where you can see the VMs that we deployed to get K8S up and running:

Step 10: Login to K8S

With Kubernetes now deployed, we can login to the main dashboard to make sure everything is working. Just point a browser at https://<ip-of-k8s-load-balancer:6443, and login with the credentials admin/admin. If everything is working, the Admin view should look something like this:

Summary

OK, there were a couple of hiccups. The issue with the UI missing network information, and the photon CLI command issue with ‘enable-cluster-type’. Both of these issues have been fed back to the team. Also, quotas take a little getting used to, but once you are familiar with them, they are far more flexible than the older ‘resource-ticket’. And if I took a little more time to learn about them, I would have had so much trial and error. Having said all that, it is still relatively straight-forward to deploy K8S onto a vSAN datastore via Photon Platform v1.2, even using the CLI approach.

11 Replies to “Deploy Kubernetes on Photon Platform 1.2 and VSAN”

  1. Thanks Cormac for educating us.

    Just wanted to post my observation.

    I am consistently seeing this behavior where the “worker” fails at creation time but the master, etcd and load-balancer are installed fine and up and running.

    Though the state of the Kubernetes-Cluster is shown as FATAL_ERROR, I am still able to login as admin and access the Kubernetes Admin Dashboard,

    The error message shows the following:

    [Task “ATTACH_ISO”: step “UPLOAD_ISO” failed with error code “InternalError” ……]

    1. Sounds like DHCP may not be working. Can you confirm that dhcp is available on the network where the workers are being provisioned?

  2. I am pretty sure DHCP is working fine. When I do a ‘tail -f /var/lib/dhcpd/dhcpd.leases’ , I do see IP addresses getting allocated from the pool to the VMs namely ETCD, MASTER, LOAD BALANCER and WORKER , when I entered the following command:

    $ photon service create -n K8S-Cluster -k KUBERNETES –dns 172.16.10.251 \
    –gateway 172.16.10.253 –netmask 255.255.255.0 \
    –master-ip 172.16.10.31 –etcd1 172.16.10.41 –load-balancer-ip 172.16.10.50 \
    –container-network 10.2.0.0/16 -c 1 –vm_flavor service-other-vm

    As I mentioned before, MASTER, ETCD and LOAD BALANCER VMs are created fine and so I can log into the Kubernetes MANAGEMENT UI. But then since the failure happened while creating the WORKER VM, there is no way (I believe) to run any workloads (like running a WordPress)

    Here is what I get:

    $ photon service create -n K8S-Cluster -k KUBERNETES –dns 172.16.10.251 –gateway 172.16.10.253 –netmask 255.255.255.0 –master-ip 172.16.10.31 –etcd1 172.16.10.41 –load-balancer-ip 172.16.10.50 –container-network 10.2.0.0/16 -c 1 –vm_flavor service-other-vm
    Service ssh key file path (leave blank for none):
    Kubernetes master 2 static IP address (leave blank for none):
    etcd server 2 static IP address (leave blank for none):

    Creating service: K8S-Cluster (KUBERNETES)
    Worker count: 1

    Are you sure [y/n]? y
    2017/06/12 16:22:57 Error: photon: Task ‘c036d4ea-a493-4fa8-b35a-983aa44b7bd7’ is in error state: {@step=={“sequence”=>”5″,”state”=>”ERROR”,”errors”=>[photon: { HTTP status: ‘0’, code: ‘InternalError’, message: ‘Failed to rollout KubernetesWorker. Error: MultiException[java.lang.IllegalStateException: VmProvisionTaskService failed for serviceID 46f575ab-964d-42a7-80a3-b6f130e50743 with error [Task “ATTACH_ISO”: step “UPLOAD_ISO” failed with error code “InternalError”, message “Request #null failed due to an internal error: com.vmware.photon.controller.api.frontend.exceptions.external.VmNotFoundException: VM 6a5a7941-ed24-49b1-809b-429b7ff3ae19 not found”]. /photon/servicesmanager/vm-provision-tasks/496851cc551cb98e9daf9]’, data: ‘map[]’ }],”warnings”=>[],”operation”=>”CREATE_KUBERNETES_SERVICE_SETUP_WORKERS”,”startedTime”=>”1497309771000″,”queuedTime”=>”1497309420000″,”endTime”=>”1497309776000″,”options”=>map[]}}
    API Errors: [photon: { HTTP status: ‘0’, code: ‘InternalError’, message: ‘Failed to rollout KubernetesWorker. Error: MultiException[java.lang.IllegalStateException: VmProvisionTaskService failed for serviceID 46f575ab-964d-42a7-80a3-b6f130e50743 with error [Task “ATTACH_ISO”: step “UPLOAD_ISO” failed with error code “InternalError”, message “Request #null failed due to an internal error: com.vmware.photon.controller.api.frontend.exceptions.external.VmNotFoundException: VM 6a5a7941-ed24-49b1-809b-429b7ff3ae19 not found”]. /photon/servicesmanager/vm-provision-tasks/496851cc551cb98e9daf9]’, data: ‘map[]’ }]
    [student@jumpbox ~]$

    1. You provide static IP addresses for the master, etcd and load-balancer. It is only the workers that use DHCP. Can you deploy a test VM, and see if it gets allocated an IP address from your DHCP pool. Remember to deply it on the VM network that you have used for K8s, and have set as the default network. I seem to remember that this is the error I got when DHCP was not working.

      1. Sorry. My bad 🙂 Yes. I am passing the static IPs of Master, etcd and load balancer. But still I don’t get why I see the MAC addresses of the same VMs (master, etcd and load balancer) showing up in /var/lib/dhcpd/dhcpd.leases’ which was meant for DHCP ?! that’s where I got confused.

        Yes. I am using the VM Network that I made as the default network.

        I will now create a test vm to make sure DHCP works fine.

        Thanks!

  3. Hi Cormac,

    Can you please let me know the hard disk size for the WORKER VM ?

    thanks!

    1. It is whatever you have defined in the VM flavor that you have used for K8s deployment. In your previous comment, it looks like it is “–vm_flavor service-other-vm”.

      1. Created a test vm. Verfied that DHCP is working fine. I am seeing the address allocated from the DHCP pool. thanks.

          1. Sure Cormac. Will do. Thanks for everything. I wouldn’t have come this far without your articles and hand-holding. Thanks again.

  4. Hello Cormac , i have photon 1.2 and deploy kubernetes 1.6 fail with FATAL_ERROR ..do you know what to check and how to fix this.

Comments are closed.