Regular readers will have seen a number of articles on this site which use the tanzu command line to create and delete TKGm clusters. TKGm is the nomenclature that I am using to describe multi-cloud TKG clusters (also known as standalone TKG clusters) that can be deployed onto numerous different IaaS, including vSphere. In this post, I want to show you how to use the same tanzu CLI tools to deploy a Tanzu Kubernetes cluster via the TKG service (TKGS) on vSphere with Tanzu. I have always shown that to deploy TKG clusters on vSphere with Tanzu, you login to the Supervisor cluster, select the appropriate namespace, populate a YAML manifest with the cluster details, and then deploy the cluster using kubectl. Here will explore an alternative way of doing it with the tanzu CLI.
A number of assumptions around pre-requisites are made in this post.
- Tanzu CLI is already installed. See official documentation for details.
- vSphere with Tanzu is installed. This post described a deployment on vSphere 7.0U3c.
- A namespace has been created. In my case, it is called workload. This can be created via the Namespace Service through the CLI or via the UI
With these requirements in place we can proceed.
Step 1: Add a user to Namespace Permissions
Ensure that there is a valid account added to the permissions tab of the Namespace, and that you use those account privileges when accessing the cluster context. I am using the administrator SSO account for simplicity, but you may wish to use something more appropriate.
Step 2: Add a Storage Class / Storage Policy to Namespace for TKG PVs
When we build the workload cluster, it will require at least one Storage Class. The Storage Class is used for the root file system disks on both the control plane nodes and the worker nodes when the cluster is created. A Storage Class is also necessary if you wish to create applications in the workload cluster that have a persistent volume requirement. In this example, I am adding two Storage Policies (both related to vSAN) to the Namespace. These Storage Policies appear as Storage Classes when queried via kubectl.
% kubectl-vsphere login --server=https://xx.xx.62.18 --vsphere-username firstname.lastname@example.org --insecure-skip-tls-verify Logged in successfully. You have access to the following contexts: xx.xx.62.18 workload If the context you wish to use is not in this list, you may need to try logging in again later, or contact your cluster administrator. % kubectl config use-context workload Switched to context "workload". % kubectl get sc NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE raid1 csi.vsphere.vmware.com Delete Immediate true 19h vsan-default-storage-policy csi.vsphere.vmware.com Delete Immediate true 19h %
Step 3: Add a VM Class to Namespace for TKG Nodes
We now need to add at least one virtual machine class to the Namespace. There are already a number of example VM classes provided that can be selected as needed, or you can also build your own VM Class. Querying VM classes via kubectl is a little strange. The command kubectl get vmclass will display any VM classes that have been assigned to any namespace on the Supervisor cluster. However, it will not show any classes that have not been assigned, so it will be difficult to see the available VM classes from the CLI in most clusters. Thus, you will definitely need to visit to vSphere client UI to see what is available, especially if this is the first namespace. In this example, I am assigning a single VM class guaranteed-small to the namespace. Once the VM class is associated with the namespace, you should be able to verify the association via the kubectl get vmclassbinding command.
% kubectl get vmclass NAME CPU MEMORY AGE guaranteed-small 2 4Gi 19h % kubectl get vmclassbinding NAME VIRTUALMACHINECLASS AGE guaranteed-small guaranteed-small 19h %
I have also added a Content Library for the VM Classes in this namespace. This step is not a requirement for this exercise.
Step 4: Select a Tanzu Kubernetes release (TKr)
The next step is to decide which release of Kubernetes to include in our cluster. The command kubectl get tkr (short for tanzukubernetesreleases) will display the list of available releases, and whether or not they are compatible. These can then be added to the cluster configuration file, along with the storage class and VM class.
% kubectl get tkr NAME VERSION READY COMPATIBLE CREATED UPDATES AVAILABLE v1.16.12---vmware.1-tkg.1.da7afe7 1.16.12+vmware.1-tkg.1.da7afe7 True True 3d23h [1.17.17+vmware.1-tkg.1.d44d45a 1.16.14+vmware.1-tkg.1.ada4837] v1.16.14---vmware.1-tkg.1.ada4837 1.16.14+vmware.1-tkg.1.ada4837 True True 3d23h [1.17.17+vmware.1-tkg.1.d44d45a] v1.16.8---vmware.1-tkg.3.60d2ffd 1.16.8+vmware.1-tkg.3.60d2ffd False False 3d23h [1.17.17+vmware.1-tkg.1.d44d45a 1.16.14+vmware.1-tkg.1.ada4837] v1.17.11---vmware.1-tkg.1.15f1e18 1.17.11+vmware.1-tkg.1.15f1e18 True True 3d23h [1.18.15+vmware.1-tkg.2.ebf6117 1.17.17+vmware.1-tkg.1.d44d45a] v1.17.11---vmware.1-tkg.2.ad3d374 1.17.11+vmware.1-tkg.2.ad3d374 True True 3d23h [1.18.15+vmware.1-tkg.2.ebf6117 1.17.17+vmware.1-tkg.1.d44d45a] v1.17.13---vmware.1-tkg.2.2c133ed 1.17.13+vmware.1-tkg.2.2c133ed True True 3d23h [1.18.15+vmware.1-tkg.2.ebf6117 1.17.17+vmware.1-tkg.1.d44d45a] v1.17.17---vmware.1-tkg.1.d44d45a 1.17.17+vmware.1-tkg.1.d44d45a True True 3d23h [1.18.15+vmware.1-tkg.2.ebf6117] v1.17.7---vmware.1-tkg.1.154236c 1.17.7+vmware.1-tkg.1.154236c True True 3d23h [1.18.15+vmware.1-tkg.2.ebf6117 1.17.17+vmware.1-tkg.1.d44d45a] v1.17.8---vmware.1-tkg.1.5417466 1.17.8+vmware.1-tkg.1.5417466 True True 3d23h [1.18.15+vmware.1-tkg.2.ebf6117 1.17.17+vmware.1-tkg.1.d44d45a] v1.18.10---vmware.1-tkg.1.3a6cd48 1.18.10+vmware.1-tkg.1.3a6cd48 True True 3d23h [1.19.7+vmware.1-tkg.2.f52f85a 1.18.15+vmware.1-tkg.2.ebf6117] v1.18.15---vmware.1-tkg.1.600e412 1.18.15+vmware.1-tkg.1.600e412 True True 3d23h [1.19.7+vmware.1-tkg.2.f52f85a 1.18.15+vmware.1-tkg.2.ebf6117] v1.18.15---vmware.1-tkg.2.ebf6117 1.18.15+vmware.1-tkg.2.ebf6117 True True 3d23h [1.19.7+vmware.1-tkg.2.f52f85a] v1.18.5---vmware.1-tkg.1.c40d30d 1.18.5+vmware.1-tkg.1.c40d30d True True 3d23h [1.19.7+vmware.1-tkg.2.f52f85a 1.18.15+vmware.1-tkg.2.ebf6117] v1.19.7---vmware.1-tkg.1.fc82c41 1.19.7+vmware.1-tkg.1.fc82c41 True True 3d23h [1.20.7+vmware.1-tkg.1.7fb9067 1.19.7+vmware.1-tkg.2.f52f85a] v1.19.7---vmware.1-tkg.2.f52f85a 1.19.7+vmware.1-tkg.2.f52f85a True True 3d23h [1.20.7+vmware.1-tkg.1.7fb9067] v1.20.2---vmware.1-tkg.1.1d4f79a 1.20.2+vmware.1-tkg.1.1d4f79a True True 3d23h [1.20.7+vmware.1-tkg.1.7fb9067] v1.20.2---vmware.1-tkg.2.3e10706 1.20.2+vmware.1-tkg.2.3e10706 True True 3d23h [1.20.7+vmware.1-tkg.1.7fb9067] v1.20.7---vmware.1-tkg.1.7fb9067 1.20.7+vmware.1-tkg.1.7fb9067 True True 3d23h
In this example, I am going to use the latest release, which is version 1.20.7.
Step 5: Create a TKG cluster configuration file
With all of the previous information gathered, I can now proceed with the creation of the configuration file for my Tanzu Kubernetes cluster. An example of a configuration file, with various comments, can be found in the official documentation here. Here is a simplified example that I am going to use to deploy my cluster. I am going to deploy a simple “dev” plan cluster made up of a single control plane node and a single worker node. I am not enabling MachineChecks or AutoScaler. I am using the same VM class and the same Storage Class for both the control plane and the worker node. I have also give the cluster a name, and set the namespace context appropriately.
% cat my-vsphere-tkc.yaml INFRASTRUCTURE_PROVIDER: tkg-service-vsphere CLUSTER_PLAN: dev CLUSTER_NAME: workload1 NAMESPACE: workload CNI: antrea CONTROL_PLANE_STORAGE_CLASS: raid1 CONTROL_PLANE_VM_CLASS: guaranteed-small SERVICE_DOMAIN: rainpole.com STORAGE_CLASSES: vsan-default-storage-policy WORKER_STORAGE_CLASS: raid1 WORKER_VM_CLASS: guaranteed-small SERVICE_CIDR: 100.64.0.0/13 CLUSTER_CIDR: 100.96.0.0/11 ENABLE_MHC: false ENABLE_AUTOSCALER: false
Step 6: Connect Supervisor context via tanzu CLI, and deploy a TKG
The next step is to use the tanzu CLI to connect to the Supervisor. My Supervisor cluster context is still set, from my login in step 1, so I just need to use the correct tanzu login syntax. Once set, I can now use the tanzu CLI to deploy the cluster, specifying the configuration file and the desired TKr.
% tanzu login --context xx.xx.62.18 ? Give the server a name xx.xx.62.18 ✔ successfully logged in to management cluster using the kubeconfig xx.xx.62.18 % tanzu cluster create --file ./my-vsphere-tkc.yaml --tkr=v1.20.7---vmware.1-tkg.1.7fb9067 You are trying to create a cluster with kubernetes version '1.20.7+vmware.1-tkg.1.7fb9067' on vSphere with Tanzu, \ Please make sure virtual machine image for the same is available in the cluster content library. Do you want to continue? [y/N]: y Validating configuration... Error: failed to apply the cluster configuration: kubectl apply failed, output: \ tanzukubernetescluster.run.tanzu.vmware.com/workload1 unchanged secret/workload1-capabilities configured secret/workload1-config-values configured Error from server (Forbidden): error when retrieving current configuration of: Resource: "addons.cluster.x-k8s.io/v1alpha3, Resource=clusterresourcesets", \ GroupVersionKind: "addons.cluster.x-k8s.io/v1alpha3, Kind=ClusterResourceSet" Name: "workload1-capabilities", Namespace: "workload" from server for: "/var/folders/lv/b1078qn55y52sq__y1533nlc0000gp/T/kubeapply-3429424587": \ clusterresourcesets.addons.cluster.x-k8s.io "workload1-capabilities" is forbidden: \ User "sso:Administrator@vsphere.local" cannot get resource "clusterresourcesets" \ in API group "addons.cluster.x-k8s.io" in the namespace "workload" : exit status 1 ✖ exit status 1
Note that the error thrown during the validation is expected. It does not prevent the cluster from being deployed. It is also documented as something which can be ignored in the TKG v1.4 Release Notes Known Issues.
Step 7: Query the status of the workload cluster
You can query the status of the cluster using various commands.
% kubectl get cluster -n workload NAME PHASE workload1 Provisioned % kubectl get tanzukubernetescluster -n workload NAME CONTROL PLANE WORKER TKR NAME AGE READY TKR COMPATIBLE UPDATES AVAILABLE workload1 1 1 v1.20.7---vmware.1-tkg.1.7fb9067 16m True True % tanzu cluster list NAME NAMESPACE STATUS CONTROLPLANE WORKERS KUBERNETES ROLES PLAN workload1 workload running 1/ 1/ 1.20.7+vmware.1-tkg.1.7fb9067 <none> % tanzu cluster get workload1 -n workload NAME NAMESPACE STATUS CONTROLPLANE WORKERS KUBERNETES ROLES workload1 workload running 1/ 1/ 1.20.7+vmware.1-tkg.1.7fb9067 <none> ℹ Details: NAME READY SEVERITY REASON SINCE MESSAGE /workload1 True 13m ├─ClusterInfrastructure - WCPCluster/workload1 True 17m ├─ControlPlane - KubeadmControlPlane/workload1-control-plane True 13m │ └─Machine/workload1-control-plane-jc5gh True 14m └─Workers └─MachineDeployment/workload1-workers-6nh9p └─Machine/workload1-workers-6nh9p-6ff9659d78-l62cw True 11m % kubectl get virtualmachines NAME POWERSTATE AGE workload1-control-plane-jc5gh poweredOn 25m workload1-workers-6nh9p-6ff9659d78-l62cw poweredOn 21m
Step 8: Access the cluster context
The cluster context can now be accessed using either the kubectl-vsphere command, or using the tanzu CLI. I have provided examples of how to do both below.
8.1 Using the tanzu CLI
In the first example, we use the tanzu CLI. I have repeated the login steps and connecting the tanzu CLI with the context, but these may not be necessary if you are still in the same context which we setup earlier.
% kubectl-vsphere login --server xx.xx.62.18 --vsphere-username email@example.com \ --insecure-skip-tls-verify Logged in successfully. You have access to the following contexts: xx.xx.62.18 workload If the context you wish to use is not in this list, you may need to try logging in again later, or contact your cluster administrator. To change context, use `kubectl config use-context <workload name>` % tanzu login --context xx.xx.62.18 ? Select a server xx.xx.62.18 () ✔ successfully logged in to management cluster using the kubeconfig xx.xx.62.18 % tanzu cluster list NAME NAMESPACE STATUS CONTROLPLANE WORKERS KUBERNETES ROLES PLAN workload1 workload running 1/ 1/ 1.20.7+vmware.1-tkg.1.7fb9067 <none> % tanzu cluster kubeconfig get workload1 -n workload --admin Credentials of cluster 'workload1' have been saved You can now access the cluster by running 'kubectl config use-context workload1-admin@workload1' % kubectl config use-context workload1-admin@workload1 Switched to context "workload1-admin@workload1". % kubectl get nodes NAME STATUS ROLES AGE VERSION workload1-control-plane-pz6p7 Ready control-plane,master 4h8m v1.20.7+vmware.1 workload1-workers-nqkvn-5c996c47f6-t4qlm Ready <none> 4h5m v1.20.7+vmware.1
8.2 Using the kubectl-vsphere command
Alternatively, you could use the original way of accessing the cluster using the kubectl-vsphere command, and selecting the namespace and cluster.
% kubectl-vsphere login --server xx.xx.62.18 --vsphere-username firstname.lastname@example.org \ --insecure-skip-tls-verify --tanzu-kubernetes-cluster-namespace workload \ --tanzu-kubernetes-cluster-name workload1 Logged in successfully. You have access to the following contexts: xx.xx.62.18 workload1 If the context you wish to use is not in this list, you may need to try logging in again later, or contact your cluster administrator. To change context, use kubectl config use-context <workload name> % kubectl config get-contexts CURRENT NAME CLUSTER AUTHINFO NAMESPACE xx.xx.62.18 xx.xx.62.18 wcp:xx.xx.62.18:email@example.com workload xx.xx.62.16 wcp:xx.xx.62.16:firstname.lastname@example.org workload * workload1 xx.xx.62.20 wcp:xx.xx.62.20:email@example.com % kubectl get nodes NAME STATUS ROLES AGE VERSION workload1-control-plane-pz6p7 Ready control-plane,master 4h3m v1.20.7+vmware.1 workload1-workers-nqkvn-5c996c47f6-t4qlm Ready <none> 4h v1.20.7+vmware.1
In conclusion, I think this is an interesting exercise to use the tanzu CLI to deploy workload clusters on vSphere with Tanzu. I guess tanzu CLI becomes more interesting if your are deploying Tanzu Kubernetes clusters to multiple IaaS, one of which is vSphere with Tanzu.
Note that Tanzu Services is now available in VMware Cloud. VMware Cloud allows vSphere administrators to deploy Tanzu Kubernetes clusters without having to manage the underlying SDDC infrastructure. Read more about Tanzu Services here.