VCP to vSphere CSI Migration in Kubernetes
When VMware first introduced support for Kubernetes, our first storage driver was the VCP, the in-tree vSphere Cloud Provider. Some might remember that this driver was referred to as Project Hatchway back in the day. This in-tree driver allows Kubernetes to consume vSphere storage for persistent volumes. One of draw-backs to the in-tree driver approach was that every storage vendor had to include their own driver in each Kubernetes distribution, which ballooned the core Kubernetes code and made maintenance difficult. Another drawback of this approach was that vendors typically had to wait for a new version of Kubernetes to release before they could patch and upgrade their own in-tree driver. This led to the creation of the Container Storage interface (CSI) specification, which essentially defined a plug-in mechanism for storage drivers, and allowed vendors to release their own updates to the storage driver as needed, which means we no longer have to wait for a new Kubernetes release for patches and updates.
The Kubernetes community has been talking about deprecating the in-tree drivers for some time, and the onus is on the various vendors to deliver a CSI driver in its place. In fact, many of the in-tree drivers are no longer maintained, so the in-tree drivers no longer get new features such as online extend, and volume snapshots. The vSphere CSI has already been available for a number of years, but the in-tree VCP is still in use. In this post, I will discuss the new vSphere CSI Migration “beta” feature, which seamlessly transfers VCP volume requests to the underlying vSphere CSI driver. This means that even when the VCP is deprecated and removed from Kubernetes, any applications using VCP volumes won’t be impacted since volume operations will be migrated to / handled by the vSphere CSI driver.
Note: After running this experiment, I noticed that a new release of the vSphere CSI driver - version
2.2.0 - has just released. Whilst CSI Migration will work with both CSI version 2.1.x and 2.2.0, it
is recommended that you use version 2.2.0 as this has some additional CSI Migration updates.
Prerequisites
There is a complete documented process to implement CSI Migration available. I will provide a step-by-step procedure here. There are a number of prerequisites to begin:
- vSphere 7.0U1 minimum
- Kubernetes v1.19 minimum
- CPI – The vSphere Cloud Provider Interface
- CSI – The vSphere Cloud Storage Interface (v2.1.x minimum, v2.2.0 preferred)
Also ensure that your local kubectl binary is at version 1.19 or above. Here is what I used in my lab.
$ kubectl version --short Client Version: v1.20.5 Server Version:v1.20.5
Deploy vSphere CSI with Migration enabled
Installation of vSphere, Kubernetes cluster and the CPI (Cloud Provider Interface) are beyond the scope of this article. The CSI driver installation also follows the standard approach for upstream Kubernetes, with one modification. The CSI controller manifest YAML needs to have the ConfigMap called internal-feature-states.csi.vsphere.vmware.com updated with the csi-migration field set to true. After making this change, deploy both the controller and node manifests as normal.
--- apiVersion: v1 data: "csi-migration": "true" # csi-migration feature is only available for vSphere 7.0U1 kind: ConfigMap metadata: name: internal-feature-states.csi.vsphere.vmware.com namespace: kube-system ---
Install an admission webhook
This webhook enables validation which prevents users from creating or updating a StorageClass with migration specific parameters which are note allowed in the vSphere CSI storage provisioner. This is because certain StorageClass parameters which were supported in the VCP are no longer supported in CSI. Scripts are provided to set the webhook up. First, there is a script to take care of creating the appropriate certificate. Here is how to download and run it, and some example output.
$ curl -O https://raw.githubusercontent.com/kubernetes-sigs/vsphere-csi-driver/release-2.1/manifests/v2.1.0/vsphere-7.0u1/vanilla/deploy/generate-signed-webhook-certs.sh $ chmod a+x ./generate-signed-webhook-certs.sh $ ./generate-signed-webhook-certs.sh creating certs in tmpdir /tmp/tmp.ChlzHFg1aF Generating RSA private key, 2048 bit long modulus .............................................+++ .......................................+++ e is 65537 (0x10001) Warning: certificates.k8s.io/v1beta1 CertificateSigningRequest is deprecated in v1.19+, unavailable in v1.22+; use certificates.k8s.io/v1 CertificateSigningRequest certificatesigningrequest.certificates.k8s.io/vsphere-webhook-svc.kube-system created NAME AGE SIGNERNAME REQUESTOR CONDITION vsphere-webhook-svc.kube-system 0s kubernetes.io/legacy-unknown kubernetes-admin Pending certificatesigningrequest.certificates.k8s.io/vsphere-webhook-svc.kube-system approved $ kubectl get csr -A NAME AGE SIGNERNAME REQUESTOR CONDITION vsphere-webhook-svc.kube-system 13s kubernetes.io/legacy-unknown kubernetes-admin Approved,Issued
An additional manifest and script are provided which create the remaining webhook objects including a webhook deployment/Pod, Service Account, Cluster Role, Role Binding, and the Service to bind with webhook pod.
$ curl -O https://raw.githubusercontent.com/kubernetes-sigs/vsphere-csi-driver/release-2.1/manifests/v2.1.0/vsphere-7.0u1/vanilla/deploy/validatingwebhook.yaml $ curl -O https://raw.githubusercontent.com/kubernetes-sigs/vsphere-csi-driver/release-2.1/manifests/v2.1.0/vsphere-7.0u1/vanilla/deploy/create-validation-webhook.sh $ chmod a+x ./create-validation-webhook.sh $ ./create-validation-webhook.sh service/vsphere-webhook-svc created validatingwebhookconfiguration.admissionregistration.k8s.io/validation.csi.vsphere.vmware.com created serviceaccount/vsphere-csi-webhook created clusterrole.rbac.authorization.k8s.io/vsphere-csi-webhook-role created clusterrolebinding.rbac.authorization.k8s.io/vsphere-csi-webhook-role-binding created deployment.apps/vsphere-csi-webhook created
These commands check that the webhook Pod and Service deployed successfully.
$ kubectl get pods -A | grep csi kube-system vsphere-csi-controller-6f7484d584-f5kcd 6/6 Running 0 3h45m kube-system vsphere-csi-node-f7pgs 3/3 Running 0 3h42m kube-system vsphere-csi-node-fpxdl 3/3 Running 0 3h42m kube-system vsphere-csi-node-lkgg5 3/3 Running 0 3h42m kube-system vsphere-csi-node-w8kt9 3/3 Running 0 3h42m kube-system vsphere-csi-webhook-7897cfb96d-9bxwf 1/1 Running 0 3h14m $ kubectl get svc vsphere-webhook-svc -n kube-system NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE vsphere-webhook-svc ClusterIP 10.106.219.128 <none> 443/TCP 3h19m $ kubectl get deploy vsphere-csi-webhook -n kube-system NAME READY UP-TO-DATE AVAILABLE AGE vsphere-csi-webhook 1/1 1 1 42s
Enabling CSI Migration on the Kubernetes Cluster
We now need to make some adjustments to the Kubernetes Cluster to support VCP -> CSI Migration. Since this is still a beta feature, we need to add some feature flags to the kube-controller on the control plane node(s) and to the kubelet on all control plane and worker/workload nodes. The feature flags are CSIMigrationvSphere and CSIMigration. The CSIMigrationvSphere flag routes volume operations from the vSphere in-tree VCP plugin to vSphere CSI plugin. If there is no vSphere CSI driver installed, the operations revert to using the in-tree VCP. The CSIMigrationvSphere feature flag requires the CSIMigration feature flag.
kube-controller changes on the control plane node(s)
Logon to the control plane node(s) and edit the /etc/kubernetes/manifests/kube-controller-manager.yaml manifest to add the feature-gates entry, highlighted in blue, at the bottom of the file below:
apiVersion: v1 kind: Pod metadata: creationTimestamp: null labels: component: kube-controller-manager tier: control-plane name: kube-controller-manager namespace: kube-system spec: containers: - command: - kube-controller-manager - --allocate-node-cidrs=true - --authentication-kubeconfig=/etc/kubernetes/controller-manager.conf - --authorization-kubeconfig=/etc/kubernetes/controller-manager.conf - --bind-address=127.0.0.1 - --client-ca-file=/etc/kubernetes/pki/ca.crt - --cloud-config=/etc/kubernetes/vsphere.conf - --cloud-provider=vsphere - --cluster-cidr=10.244.0.0/16 - --cluster-name=kubernetes - --cluster-signing-cert-file=/etc/kubernetes/pki/ca.crt - --cluster-signing-key-file=/etc/kubernetes/pki/ca.key - --controllers=*,bootstrapsigner,tokencleaner - --kubeconfig=/etc/kubernetes/controller-manager.conf - --leader-elect=true - --port=0 - --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt - --root-ca-file=/etc/kubernetes/pki/ca.crt - --service-account-private-key-file=/etc/kubernetes/pki/sa.key - --service-cluster-ip-range=10.96.0.0/12 - --use-service-account-credentials=true - --feature-gates=CSIMigration=true,CSIMigrationvSphere=true image: k8s.gcr.io/kube-controller-manager:v1.20.5
kubelet config changes on the control plane node(s)
Logon to the control plane node(s) and edit the kubelet /var/lib/kubelet/config.yaml manifest to add the feature-gates entries, highlighted in blue, at the bottom of the file below:
apiVersion: kubelet.config.k8s.io/v1beta1 authentication: anonymous: enabled: false webhook: cacheTTL: 0s enabled: true x509: clientCAFile: /etc/kubernetes/pki/ca.crt authorization: mode: Webhook webhook: cacheAuthorizedTTL: 0s cacheUnauthorizedTTL: 0s cgroupDriver: systemd clusterDNS: - 10.96.0.10 clusterDomain: cluster.local cpuManagerReconcilePeriod: 0s evictionPressureTransitionPeriod: 0s fileCheckFrequency: 0s healthzBindAddress: 127.0.0.1 healthzPort: 10248 httpCheckFrequency: 0s imageMinimumGCAge: 0s kind: KubeletConfiguration logging: {} nodeStatusReportFrequency: 0s nodeStatusUpdateFrequency: 0s resolvConf: /run/systemd/resolve/resolv.conf rotateCertificates: true runtimeRequestTimeout: 0s shutdownGracePeriod: 0s shutdownGracePeriodCriticalPods: 0s staticPodPath: /etc/kubernetes/manifests streamingConnectionIdleTimeout: 0s syncFrequency: 0s volumeStatsAggPeriod: 0s featureGates: CSIMigration: true CSIMigrationvSphere: true
After making the changes, the kubelet should be restarted on the control plane node(s) as follows. It is advisable to check on the status of the kubelet afterwards to make sure it starts. If it fails to restart, it may be due to a typo you placed in the configuration file. A command such as Ubuntu’s journalctl -xe -u kubelet can be used to check for reasons why the kubelet did not restart successfully.
$ sudo systemctl restart kubelet $ sudo systemctl status kubelet ● kubelet.service - kubelet: The Kubernetes Node Agent Loaded: loaded (/lib/systemd/system/kubelet.service; enabled; vendor preset: enabled) Drop-In: /etc/systemd/system/kubelet.service.d └─10-kubeadm.conf Active: active (running) since Tue 2021-04-13 09:26:47 UTC; 1s ago Docs: https://kubernetes.io/docs/home/ Main PID: 3848205 (kubelet) Tasks: 10 (limit: 4676) Memory: 22.3M CGroup: /system.slice/kubelet.service └─3848205 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --cloud-config=/etc/kubernetes/vsphere.co>
This will trigger a restart of the control plane services in kube-system. Use the kubectl get pods -A command to monitor etcd, api-server and controller manager. These should be Running and Ready 1/1 within a few minutes. Wait for everything to be available before proceeding with further changes.
kubelet config changes on the worker node(s)
This procedure for the worker nodes is the same as what we did on the control plane nodes, except that before we make the changes, we need to drain the worker nodes of any running applications. Once the worker node has been drained, logon to the node, change the feature gates on the kubelet config, restart the kubelet, logout, then uncordon the node so that applications can once again be scheduled on it. The feature gate entries for the worker node kubelet config are exactly the same as the feature gate entries on the control nodes.
$ kubectl drain k8s-worker-01 --force --ignore-daemonsets node/k8s-worker-01 cordoned WARNING: ignoring DaemonSet-managed Pods: kube-system/kube-flannel-ds-r899q, kube-system/kube-proxy-dvktz, kube-system/vsphere-csi-node-lkgg5 node/k8s-worker-01 drained $ ssh ubuntu@k8s-worker-01 ubuntu@k8s-worker-01:~$ sudo vi /var/lib/kubelet/config.yaml <<-- add feature gates entries and save the file -->> featureGates: CSIMigration: true CSIMigrationvSphere: true ubuntu@k8s-worker-01:~$ sudo systemctl restart kubelet ubuntu@k8s-worker-01:~$ sudo systemctl status kubelet ubuntu@k8s-worker-01:~$ exit $ kubectl uncordon k8s-worker-01 node/k8s-worker-01 uncordoned
Repeat this process for all worker nodes. Again, if there are issues with the edits you made to the config file, you can use a command similar to Ubuntu’s journalctl -xe -u kubelet to examine the logs and see what is causing the error.
Migration Annotations
We should now be able to check the annotations on nodes, Pods, PVCs and PVs to verify that these are indeed migrated to use the CSI driver.
CSINode Migration Annotations
Each CSINode object should now display a migrated-plugins annotation, as shown here.
$ kubectl describe csinodes k8s-controlplane-01 | grep Annot Annotations: storage.alpha.kubernetes.io/migrated-plugins: kubernetes.io/vsphere-volume $ kubectl describe csinodes k8s-worker-03 | grep Annot Annotations: storage.alpha.kubernetes.io/migrated-plugins: kubernetes.io/vsphere-volume $ kubectl describe csinodes k8s-worker-02 | grep Annot Annotations: storage.alpha.kubernetes.io/migrated-plugins: kubernetes.io/vsphere-volume $ kubectl describe csinodes k8s-worker-01 | grep Annot Annotations: storage.alpha.kubernetes.io/migrated-plugins: kubernetes.io/vsphere-volume
PVC Migration Annotations
In this Kubernetes cluster, I had a previously deployed an application (a NoSQL Cassandra DB). It is deployed as a StatefulSet with 3 replicas, thus 3 Pods and 3 PVCs and 3 PVs. It was using the VCP when originally deployed. If I check the Persistent Volume Claims (PVCs) of this application, I should now see a new migration annotation, as highlighted in blue below.
$ kubectl describe pvc cassandra-data-cassandra-0 -n cassandra Name: cassandra-data-cassandra-0 Namespace: cassandra StorageClass: cass-sc-vcp Status: Bound Volume: pvc-900b85d6-c2a8-4996-a7a4-ec9a23bffd77 Labels: app=cassandra Annotations: pv.kubernetes.io/bind-completed: yes pv.kubernetes.io/bound-by-controller: yes pv.kubernetes.io/migrated-to: csi.vsphere.vmware.com volume.beta.kubernetes.io/storage-provisioner: kubernetes.io/vsphere-volume Finalizers: [kubernetes.io/pvc-protection] Capacity: 1Gi Access Modes: RWO VolumeMode: Filesystem Used By: cassandra-0 Events: <none>
PV Migration Annotations
A similar annotation should also be associated with the Persistent Volumes used by the StatefulSet.
$ kubectl describe pv pvc-affa795f-db69-4456-8baa-6a2fe9f19d2e -n cassandra Name: pvc-affa795f-db69-4456-8baa-6a2fe9f19d2e Labels: <none> Annotations: kubernetes.io/createdby: vsphere-volume-dynamic-provisioner pv.kubernetes.io/bound-by-controller: yes pv.kubernetes.io/migrated-to: csi.vsphere.vmware.com pv.kubernetes.io/provisioned-by: kubernetes.io/vsphere-volume Finalizers: [kubernetes.io/pv-protection external-attacher/csi-vsphere-vmware-com] StorageClass: cass-sc-vcp Status: Bound Claim: cassandra/cassandra-data-cassandra-2 Reclaim Policy: Delete Access Modes: RWO VolumeMode: Filesystem Capacity: 1Gi Node Affinity: <none> Message: Source: Type: vSphereVolume (a Persistent Disk resource in vSphere) VolumePath: [vsanDatastore-OCTO-Cluster-B] d85b7460-17a6-6ea1-4b39-246e962f497c/kubernetes-dynamic-pvc-affa795f-db69-4456-8baa-6a2fe9f19d2e.vmdk FSType: ext4 StoragePolicyName: vsan-b Events: <none>
Note that the PV specification continues to keep the original VCP Volume Path. If there is ever any reason to disable the CSI migration feature, the volume operations can revert to using the in-tree VCP plugin.
Caution: During my testing, I found that a limitation on the CSI Volume ID which limits the number of characters in the volumes ID path to 128. Thus, if you have a vSphere datastore name that is longer than 24 characters, you may hit an issue with migrating pre-existing VCP volumes to CSI. This has been reported as a Kubernetes issue. By example, here is a test with different length datastore names: (CSI migration works - volume id length < 128 ) VolumePath: [vsan-OCTO-Cluster-B] d85b7460-17a6-6ea1-4b39-246e962f497c/ kubernetes-dynamic-pvc-920d248d-7e66-451a-a291-c7dfa3458a72.vmdk (CSI migration fails - volume id length > 128) VolumePath: [vsanDatastore-OCTO-Cluster-B] d85b7460-17a6-6ea1-4b39-246e962f497c/ kubernetes-dynamic-pvc-91c5826f-ed8e-4d55-a5bc-c4b31e435f5d.vmdk
Functionality test
Since it is now the CSI driver that is doing the volume provisioning, we are no longer able to use StorageClass parameters that we were able to use with the VCP. For example, if we now try to provision a volume that has the StorageClass parameter diskformat: eagerzeroedthick, which is supported with the VCP but not the CSI, the volume provisioning will fail (as described in the webhook section earlier).
$ cat vsan-sc-vcp.yaml kind: StorageClass apiVersion: storage.k8s.io/v1 metadata: name: vsan-sc-vcp provisioner: kubernetes.io/vsphere-volume parameters: diskformat: eagerzeroedthick storagePolicyName: "vsan-b" datastore: "vsanDatastore-OCTO-Cluster-B”
The following will be observed if you use kubectl to describe the PVC and check the events; eagerzeroedthick is no longer a valid diskformat parameter.
Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal ExternalProvisioning <invalid> (x2 over <invalid>) persistentvolume-controller waiting for a volume to be created, either by external provisioner "csi.vsphere.vmware.com" or manually created by system administrator Normal Provisioning <invalid> (x4 over <invalid>) csi.vsphere.vmware.com_vsphere-csi-controller-6f7484d584-f5kcd_f0b90478-2001-4217-b5f2-34cd53ef4c86 External provisioner is provisioning volume for claim "default/vsan-pvc-vcp" Warning ProvisioningFailed <invalid> (x4 over <invalid>) csi.vsphere.vmware.com_vsphere-csi-controller-6f7484d584-f5kcd_f0b90478-2001-4217-b5f2-34cd53ef4c86 failed to provision volume with StorageClass "vsan-sc-vcp": rpc error: code = InvalidArgument desc = Parsing storage class parameters failed with error: invalid parameter. key:diskformat-migrationparam, value:eagerzeroedthick
Hopefully this has provided you with some good guidance on how to implement VCP->CSI Migration. Note that this feature is currently beta, so is not yet permanently enabled in upstream Kubernetes distributions. However the feature flags can be enabled as shown here to allow you to test the procedure and verify that in-tree VCP volumes can now be managed with the vSphere CSI driver. Keep in mind that eventually the VCP will become deprecated and will no longer be available in Kubernetes distribution. This migration mechanism should alleviate any concerns around that deprecation.
Added benefit: VCP volumes now visible in CNS
Since VCP volumes operations are now handled by the vSphere CSI driver, the Cloud Native Storage (CNS) component on vSphere can now bubble up volume information to the vSphere client. The volumes for my NoSQL Cassandra StatefulSet, which I had previously deployed using the in-tree VCP, are now visible in the vSphere client. Here are a few screenshots to show this visibility. First, we can see the 3 persistent volumes.
Next, if we click on the details icon, we can see some basic information about the volume, including which datastore it is provisioned onto:
If we select the Kubernetes objects view, we can see more detail about the K8s cluster and volume. This is vanilla Kubernetes, not one of the VMware Tanzu editions. We can also see the K8s namespace and any Pod(s) that are using the volume.
Lastly, we can see the physical placement. In this case, since it is vSAN datastore, we can see exactly how this volume is built (it is a RAID-1). So we have end-to-end visibility from K8s volume to vSphere volume to physical storage device.
Pretty neat!