A closer look at the v1alpha2 TanzuKubernetesCluster format in vSphere with Tanzu

Today I wanted to take a closer look at the new Tanzu Kubernetes Cluster YAML format (v1alpha2) which extends the configurability of TKG clusters that are deployed via the TKG Service (TKGS) in vSphere with Tanzu. We will look at this from two viewpoints. The first is to show you the differences when it comes to creating a new TKG cluster, as there are a number of different manifest settings now required with the v1alpha2 format. The second viewpoint is to look at how to upgrade the Tanzu Kubernetes Release (tkr) on an existing cluster which has been upgraded from the v1alpha1 to the v1alpha2 format. The procedure is a bit different to how we upgraded v1alpha1 clusters in the past.  This format upgrade to v1alpha2 happens automatically when the Supervisor Cluster is upgraded to a version that supports the v1alpha2 format (e.g. v1.21.0), but the actual release or version of Kubernetes running in the cluster still has to be manually updated.

Creating a new v1alpha2 format TKG cluster

The easiest way to demonstrate the changes is to show the manifests for a v1alpha2 format, and compare it to the older v1alpha1 format. I have put some simple examples of manifests side by side below:

v1alpha2 format v1alpha1 format
apiVersion: run.tanzu.vmware.com/v1alpha2
kind: TanzuKubernetesCluster
metadata:
 name: tkg-cluster-01
 namespace: devops1
spec:
 topology:
  controlPlane:
    replicas: 1
    vmClass: guaranteed-small
    storageClass: vsan-default-storage-policy
    tkr:
      reference:
        name: v1.20.7---vmware.1-tkg.1.7fb9067
  nodePools:
  - name: worker-pool-1
    replicas: 2
    vmClass: guaranteed-small
    storageClass: vsan-default-storage-policy
    tkr:
      reference:
        name: v1.20.7---vmware.1-tkg.1.7fb9067
apiVersion: run.tanzu.vmware.com/v1alpha1
kind: TanzuKubernetesCluster
metadata:
 name: tkg-cluster-02
spec:
 topology:
  controlPlane:
    count: 1
    class: guaranteed-small
    storageClass: vsan-default-storage-policy
  workers:
    count: 2
    class: guaranteed-small
    storageClass: vsan-default-storage-policy
 distribution:
  version: v1.20.7

Some of the major differences to highlight are:

  • apiVersion: v1alpha2 replaces v1alpha1
  • ability to specify a namespace in the metadata
  • spec.topology.controlPlane.replicas replaces spec.topology.controlPlane.count
  • vmClass replaces class for virtual machine class type
  • A new spec.topology.controlPlane.tkr entry, which specifies the release/distribution
  • spec.topology.workers replaces by spec.topology.nodePools
  • The deprecation of distribution.version

Note that in the current release, the tkr.reference.name fields must match in both the controlPlane and in the nodePools sections. In the future, different Tanzu Kubernetes releases for node pools may be supported, so some of this new format is future-proofing.

Before you can deploy a new TKC via the TKG Service, the Virtual Machine Class and Storage Class that are specified in the cluster manifest must be added to the namespace where the cluster is being deployed. Here is a view of the “devops1” namespace in my environment, where both the VM Class and StorageClass have been added or bound already.

The command line can also be used to validate that the parameters that we wish to use to create the cluster are available. All available virtual machine classes, storage classes and the virtual machine classes bound to the namespace can be displayed, as follows:

% kubectl get virtualmachineclass
NAME                  CPU   MEMORY   AGE
best-effort-2xlarge   8     64Gi     81d
best-effort-4xlarge   16    128Gi    81d
best-effort-8xlarge   32    128Gi    81d
best-effort-large     4     16Gi     81d
best-effort-medium    2     8Gi      231d
best-effort-small     2     4Gi      231d
best-effort-xlarge    4     32Gi     81d
best-effort-xsmall    2     2Gi      81d
guaranteed-2xlarge    8     64Gi     81d
guaranteed-4xlarge    16    128Gi    81d
guaranteed-8xlarge    32    128Gi    81d
guaranteed-large      4     16Gi     231d
guaranteed-medium     2     8Gi      231d
guaranteed-small      2     4Gi      81d
guaranteed-xlarge     4     32Gi     81d
guaranteed-xsmall     2     2Gi      81d


% kubectl get storageclass
NAME                                    PROVISIONER              RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
vsan-default-storage-policy             csi.vsphere.vmware.com   Delete          Immediate           true                   253d


% kubectl get vmclassbinding -n devops1
NAME               VIRTUALMACHINECLASS   AGE
guaranteed-small   guaranteed-small      2m17s

The available Tanzu Kubernetes Releases can also be queried using the tanzukubernetesrelease directive, or tkr for short. As we can see, the latest version is now v1.20.7, which is what we placed in the manifest above.

% kubectl get tkr
NAME                                VERSION                          READY   COMPATIBLE   CREATED   UPDATES AVAILABLE
v1.16.12---vmware.1-tkg.1.da7afe7   1.16.12+vmware.1-tkg.1.da7afe7   True    True         22h       [1.17.17+vmware.1-tkg.1.d44d45a 1.16.14+vmware.1-tkg.1.ada4837]
v1.16.14---vmware.1-tkg.1.ada4837   1.16.14+vmware.1-tkg.1.ada4837   True    True         22h       [1.17.17+vmware.1-tkg.1.d44d45a]
v1.16.8---vmware.1-tkg.3.60d2ffd    1.16.8+vmware.1-tkg.3.60d2ffd    False   False        22h       [1.17.17+vmware.1-tkg.1.d44d45a 1.16.14+vmware.1-tkg.1.ada4837]
v1.17.11---vmware.1-tkg.1.15f1e18   1.17.11+vmware.1-tkg.1.15f1e18   True    True         22h       [1.18.15+vmware.1-tkg.2.ebf6117 1.17.17+vmware.1-tkg.1.d44d45a]
v1.17.11---vmware.1-tkg.2.ad3d374   1.17.11+vmware.1-tkg.2.ad3d374   True    True         22h       [1.18.15+vmware.1-tkg.2.ebf6117 1.17.17+vmware.1-tkg.1.d44d45a]
v1.17.13---vmware.1-tkg.2.2c133ed   1.17.13+vmware.1-tkg.2.2c133ed   True    True         22h       [1.18.15+vmware.1-tkg.2.ebf6117 1.17.17+vmware.1-tkg.1.d44d45a]
v1.17.17---vmware.1-tkg.1.d44d45a   1.17.17+vmware.1-tkg.1.d44d45a   True    True         22h       [1.18.15+vmware.1-tkg.2.ebf6117]
v1.17.7---vmware.1-tkg.1.154236c    1.17.7+vmware.1-tkg.1.154236c    True    True         22h       [1.18.15+vmware.1-tkg.2.ebf6117 1.17.17+vmware.1-tkg.1.d44d45a]
v1.17.8---vmware.1-tkg.1.5417466    1.17.8+vmware.1-tkg.1.5417466    True    True         22h       [1.18.15+vmware.1-tkg.2.ebf6117 1.17.17+vmware.1-tkg.1.d44d45a]
v1.18.10---vmware.1-tkg.1.3a6cd48   1.18.10+vmware.1-tkg.1.3a6cd48   True    True         22h       [1.19.7+vmware.1-tkg.2.f52f85a 1.18.15+vmware.1-tkg.2.ebf6117]
v1.18.15---vmware.1-tkg.1.600e412   1.18.15+vmware.1-tkg.1.600e412   True    True         22h       [1.19.7+vmware.1-tkg.2.f52f85a 1.18.15+vmware.1-tkg.2.ebf6117]
v1.18.15---vmware.1-tkg.2.ebf6117   1.18.15+vmware.1-tkg.2.ebf6117   True    True         22h       [1.19.7+vmware.1-tkg.2.f52f85a]
v1.18.5---vmware.1-tkg.1.c40d30d    1.18.5+vmware.1-tkg.1.c40d30d    True    True         22h       [1.19.7+vmware.1-tkg.2.f52f85a 1.18.15+vmware.1-tkg.2.ebf6117]
v1.19.7---vmware.1-tkg.1.fc82c41    1.19.7+vmware.1-tkg.1.fc82c41    True    True         22h       [1.20.7+vmware.1-tkg.1.7fb9067 1.19.7+vmware.1-tkg.2.f52f85a]
v1.19.7---vmware.1-tkg.2.f52f85a    1.19.7+vmware.1-tkg.2.f52f85a    True    True         22h       [1.20.7+vmware.1-tkg.1.7fb9067]
v1.20.2---vmware.1-tkg.1.1d4f79a    1.20.2+vmware.1-tkg.1.1d4f79a    True    True         22h       [1.20.7+vmware.1-tkg.1.7fb9067]
v1.20.2---vmware.1-tkg.2.3e10706    1.20.2+vmware.1-tkg.2.3e10706    True    True         22h       [1.20.7+vmware.1-tkg.1.7fb9067]
v1.20.7---vmware.1-tkg.1.7fb9067    1.20.7+vmware.1-tkg.1.7fb9067    True    True         22h

To create a new cluster, simply run the kubectl apply -f command that we used before, and if all goes well, the cluster should get created.

% kubectl apply -f tanzucluster-v1alpha2-v1.20.7.yaml
tanzukubernetescluster.run.tanzu.vmware.com/tkg-cluster-03 created

% kubectl get tanzukubernetesclusters -n devops1
NAME            CONTROL PLANE  WORKER  TKR NAME                          AGE  READY  TKR COMPATIBLE  UPDATES AVAILABLE
tkg-cluster-03  1              2        v1.20.7---vmware.1-tkg.1.7fb9067  37m  True    True

OK – so that demonstrates how to create a new cluster. What about upgrading an existing cluster?

Upgrading a v1alpha2 format TKG cluster

After upgrading the Supervisor Cluster in a vSphere with Tanzu environment, the TKG clusters are automatically upgraded to the v1alpha2 format. In my environment, I upgraded vSphere to version 7.0U3c and the Supervisor Cluster to version 1.21.0. However, this process does not upgrade the release version (tkr). This is still a manual step, but the fields that need to be updated are now different when compared to previous upgrades of a v1alpha1 format cluster. In a v1alpha1 cluster, you would change the following fields from:

spec:
  distribution:
    fullVersion: v1.20.2+vmware.1-tkg.2.3e10706
    version: v1.20.2

to something like:

spec:
  distribution:
    fullVersion: null
    version: v1.20.7

And this would automatically trigger a rolling update of the TKG cluster to the new version. Note that this is the procedure to follow if the Supervisor Cluster has not been upgraded to v1.21.0 and the TKG clusters have not been converted to v1alpha2 format, but are still at a v1alpha1 format.

With the new v1alpha2 format, you need to change the tkr.reference.name fields in both the controlPlane and in the nodePools section. So let’s say I wanted to upgrade my TKG cluster to v1.20.7. After running the kubectl edit tanzukubernetescluster command, I would have to change the following fields (some other fields are truncated):

.
.
topology:
    controlPlane:
      replicas: 1
      storageClass: vsan-default-storage-policy
     tkr:
        reference:
          name: v1.20.7---vmware.1-tkg.1.7fb9067         <<< change here 
      vmClass: guaranteed-small
    nodePools:
    - name: workers
      replicas: 3
      storageClass: vsan-default-storage-policy
      tkr:
        reference:
          name: v1.20.7---vmware.1-tkg.1.7fb9067         <<< and here
      vmClass: best-effort-small

This would then trigger the rolling upgrade as before. There are some nuances around this, as you can remove tkr references from some areas, and also fall back to using the deprecated distribution.version option if you wish. There is a great write-up in the official docs around how to do this if you are interested.