Requirements, steps to enable Volume Snapshot and Restore as well as configuration parameters can be found here. The official docs include a link to a bash script which checks that the snapshot feature has been enabled, pulls the necessary manifest to install the necessary snapshot components, deploys the snapshot controller, and finally updates the CSI driver with the additional snapshot sidecar. The script is available here. Here is a sample output from my Kubernetes cluster when running the deployment bash script.
% bash deploy-csi-snapshot-components.sh
No existing snapshot-controller Deployment found, deploying it now..
Start snapshot-controller deployment...
customresourcedefinition.apiextensions.k8s.io/volumesnapshotclasses.snapshot.storage.k8s.io created
Created CRD volumesnapshotclasses.snapshot.storage.k8s.io
customresourcedefinition.apiextensions.k8s.io/volumesnapshotcontents.snapshot.storage.k8s.io created
Created CRD volumesnapshotcontents.snapshot.storage.k8s.io
customresourcedefinition.apiextensions.k8s.io/volumesnapshots.snapshot.storage.k8s.io created
Created CRD volumesnapshots.snapshot.storage.k8s.io
✅ Deployed VolumeSnapshot CRDs
serviceaccount/snapshot-controller unchanged
clusterrole.rbac.authorization.k8s.io/snapshot-controller-runner unchanged
clusterrolebinding.rbac.authorization.k8s.io/snapshot-controller-role unchanged
role.rbac.authorization.k8s.io/snapshot-controller-leaderelection unchanged
rolebinding.rbac.authorization.k8s.io/snapshot-controller-leaderelection unchanged
✅ Created RBACs for snapshot-controller
deployment.apps/snapshot-controller created
deployment.apps/snapshot-controller image updated
deployment.apps/snapshot-controller patched
deployment.apps/snapshot-controller patched
Waiting for deployment spec update to be observed...
Waiting for deployment "snapshot-controller" rollout to finish: 0 out of 2 new replicas have been updated...
Waiting for deployment "snapshot-controller" rollout to finish: 1 out of 2 new replicas have been updated...
Waiting for deployment "snapshot-controller" rollout to finish: 1 out of 2 new replicas have been updated...
Waiting for deployment "snapshot-controller" rollout to finish: 1 out of 2 new replicas have been updated...
Waiting for deployment "snapshot-controller" rollout to finish: 1 out of 2 new replicas have been updated...
Waiting for deployment "snapshot-controller" rollout to finish: 1 out of 2 new replicas have been updated...
Waiting for deployment "snapshot-controller" rollout to finish: 1 out of 2 new replicas have been updated...
Waiting for deployment "snapshot-controller" rollout to finish: 1 out of 2 new replicas have been updated...
Waiting for deployment "snapshot-controller" rollout to finish: 1 of 2 updated replicas are available...
Waiting for deployment "snapshot-controller" rollout to finish: 1 of 2 updated replicas are available...
deployment "snapshot-controller" successfully rolled out
✅ Successfully deployed snapshot-controller
No existing snapshot-validation-deployment Deployment found, deploying it now..
creating certs in tmpdir /var/folders/31/y77ywvzd6lqc0g60r4xnfyd80000gp/T/tmp.HmdOwrGg7f
Generating a 2048 bit RSA private key
.............................................................................................+++
.........................+++
writing new private key to '/var/folders/31/y77ywvzd6lqc0g60r4xnfyd80000gp/T/tmp.HmdOwrGg7f/ca.key'
-----
Generating RSA private key, 2048 bit long modulus
...............................................................+++
.............................................................................+++
e is 65537 (0x10001)
Signature ok
subject=/CN=snapshot-validation-service.kube-system.svc
Getting CA Private Key
secret "snapshot-webhook-certs" deleted
secret/snapshot-webhook-certs created
service "snapshot-validation-service" deleted
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 2238 100 2238 0 0 9060 0 --:--:-- --:--:-- --:--:-- 9024
service/snapshot-validation-service created
validatingwebhookconfiguration.admissionregistration.k8s.io/validation-webhook.snapshot.storage.k8s.io configured
deployment.apps/snapshot-validation-deployment created
deployment.apps/snapshot-validation-deployment image updated
deployment.apps/snapshot-validation-deployment patched
Waiting for deployment spec update to be observed...
Waiting for deployment spec update to be observed...
Waiting for deployment "snapshot-validation-deployment" rollout to finish: 0 out of 3 new replicas have been updated...
Waiting for deployment "snapshot-validation-deployment" rollout to finish: 0 out of 3 new replicas have been updated...
Waiting for deployment "snapshot-validation-deployment" rollout to finish: 1 out of 3 new replicas have been updated...
Waiting for deployment "snapshot-validation-deployment" rollout to finish: 1 out of 3 new replicas have been updated...
Waiting for deployment "snapshot-validation-deployment" rollout to finish: 1 out of 3 new replicas have been updated...
Waiting for deployment "snapshot-validation-deployment" rollout to finish: 1 out of 3 new replicas have been updated...
Waiting for deployment "snapshot-validation-deployment" rollout to finish: 1 out of 3 new replicas have been updated...
Waiting for deployment "snapshot-validation-deployment" rollout to finish: 2 out of 3 new replicas have been updated...
Waiting for deployment "snapshot-validation-deployment" rollout to finish: 2 out of 3 new replicas have been updated...
Waiting for deployment "snapshot-validation-deployment" rollout to finish: 2 out of 3 new replicas have been updated...
Waiting for deployment "snapshot-validation-deployment" rollout to finish: 2 out of 3 new replicas have been updated...
Waiting for deployment "snapshot-validation-deployment" rollout to finish: 1 old replicas are pending termination...
Waiting for deployment "snapshot-validation-deployment" rollout to finish: 1 old replicas are pending termination...
deployment "snapshot-validation-deployment" successfully rolled out
✅ Successfully deployed snapshot-validation-deployment
csi-snapshotter side-car not found in vSphere CSI Driver Deployment, patching..
creating patch file in tmpdir /var/folders/31/y77ywvzd6lqc0g60r4xnfyd80000gp/T/tmp.GiMDimfZdq
Scale down the vSphere CSI driver
deployment.apps/vsphere-csi-controller scaled
Patching vSphere CSI driver..
deployment.apps/vsphere-csi-controller patched
Scaling the vSphere CSI driver back to original state..
deployment.apps/vsphere-csi-controller scaled
Waiting for deployment spec update to be observed...
Waiting for deployment spec update to be observed...
Waiting for deployment "vsphere-csi-controller" rollout to finish: 0 out of 3 new replicas have been updated...
Waiting for deployment "vsphere-csi-controller" rollout to finish: 0 of 3 updated replicas are available...
Waiting for deployment "vsphere-csi-controller" rollout to finish: 1 of 3 updated replicas are available...
Waiting for deployment "vsphere-csi-controller" rollout to finish: 2 of 3 updated replicas are available...
deployment "vsphere-csi-controller" successfully rolled out
✅ Successfully deployed all components for CSI Snapshot feature!
If the update has been successful, you should observe 7 containers in the vSphere CSI controller pod.
% kubectl get pod -n vmware-system-csi vsphere-csi-controller-5b6dfc6799-lmphc NAME READY STATUS RESTARTS AGE vsphere-csi-controller-5b6dfc6799-lmphc 7/7 Running 7 (12d ago) 20d
In this post, I will take an existing StorageClass, PVC, PV and pod deployment. I will then snapshot the volume, convert the snapshot to a new volume, and then mount the snapshot’ed volume to another pod. Lastly, I will verify that the data on the volume is present and available.
Here is the current test app deployment. I have a pod, PVC, PV and StorageClass already built. The pod is a very simple busybox pod which has some basic Unix utilities incorporated.
% kubectl get sc,pod,pvc,pv NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE storageclass.storage.k8s.io/csisnaps-sc (default) csi.vsphere.vmware.com Delete Immediate true 10s NAME READY STATUS RESTARTS AGE pod/csisnaps-pod 1/1 Running 0 9s NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE persistentvolumeclaim/csisnaps-pvc-vsan-claim Bound pvc-ca1960f2-6a86-43d6-80e4-6aab0f506e7a 2Gi RWO csisnaps-sc 9s NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE persistentvolume/pvc-ca1960f2-6a86-43d6-80e4-6aab0f506e7a 2Gi RWO Delete Bound default/csisnaps-pvc-vsan-claim csisnaps-sc 9s
Let’s add some simple data to the volume mounted to the busybox pod by ‘exec’ing into it and creating a directory and a file with some contents.
% kubectl exec -it csisnaps-pod -- sh / # df -h Filesystem Size Used Available Use% Mounted on overlay 77.7G 12.1G 61.6G 16% / tmpfs 64.0M 0 64.0M 0% /dev /dev/sdb 1.9G 6.0M 1.8G 0% /demo /dev/sda3 77.7G 12.1G 61.6G 16% /dev/termination-log /dev/sda3 77.7G 12.1G 61.6G 16% /etc/resolv.conf /dev/sda3 77.7G 12.1G 61.6G 16% /etc/hostname /dev/sda3 77.7G 12.1G 61.6G 16% /etc/hosts shm 64.0M 0 64.0M 0% /dev/shm tmpfs 15.5G 12.0K 15.5G 0% /var/run/secrets/kubernetes.io/serviceaccount tmpfs 7.8G 0 7.8G 0% /proc/acpi tmpfs 64.0M 0 64.0M 0% /proc/kcore tmpfs 64.0M 0 64.0M 0% /proc/keys tmpfs 64.0M 0 64.0M 0% /proc/timer_list tmpfs 7.8G 0 7.8G 0% /proc/scsi tmpfs 7.8G 0 7.8G 0% /sys/firmware / # cd /demo /demo # ls lost+found /demo # mkdir Special-Data /demo # cd Special-Data/ /demo/Special-Data # echo "my-special-data" >> special-data-file /demo/Special-Data # cat special-data-file my-special-data /demo/Special-Data #
Now we will take a snapshot of this volume. These are the manifest that I will use to first make a CSI snapshot class which defines which provider should be used to create snapshots, and what to do when a snapshot is deleted. The second manifest is used to make a snapshot. Note that the snapshot manifest contains a spec.source.persistentVolumeClaimName which determines which volume should be snapshot’ed. In this examples, it matches the PVC called csisnaps-pvc-vsan-claim, as shown above.
% cat csi-snapshot-class.yaml apiVersion: snapshot.storage.k8s.io/v1 kind: VolumeSnapshotClass metadata: name: block-snapshotclass driver: csi.vsphere.vmware.com deletionPolicy: Delete % cat dynamic-vol-snap.yaml apiVersion: snapshot.storage.k8s.io/v1 kind: VolumeSnapshot metadata: name: block-snapshotvol spec: volumeSnapshotClassName: block-snapshotclass source: persistentVolumeClaimName: csisnaps-pvc-vsan-claim
Let’s now apply the manifests, and then check the VolumeSnapshot and the VolumeSnapshotContents.
% kubectl apply -f dynamic-vol-snap.yaml volumesnapshot.snapshot.storage.k8s.io/block-snapshotvol created % kubectl get volumesnapshot NAME READYTOUSE SOURCEPVC SOURCESNAPSHOTCONTENT RESTORESIZE SNAPSHOTCLASS SNAPSHOTCONTENT CREATIONTIME AGE block-snapshotvol true csisnaps-pvc-vsan-claim 2Gi block-snapshotclass snapcontent-2471e409-7e7e-4fed-8d7a-95a73b097abf 24s 30s % kubectl get volumesnapshotcontents NAME READYTOUSE RESTORESIZE DELETIONPOLICY DRIVER VOLUMESNAPSHOTCLASS VOLUMESNAPSHOT VOLUMESNAPSHOTNAMESPACE AGE snapcontent-2471e409-7e7e-4fed-8d7a-95a73b097abf true 2147483648 Delete csi.vsphere.vmware.com block-snapshotclass block-snapshotvol default 34s
Great! It looks like a snapshot of the volume used by my simple application has been taken successfully. The next step is to restore that snapshot which will instantiate a new volume. Once that is successful, the volume can then be mounted to a pod where we can check its contents to ensure the volume data was indeed captured by the snapshot. First, I have to construct a snapshot restore manifest which is basically a PVC that includes a spec.datasource.kind set to VolumeSnapshot, along with the name of the snapshot we wish to instantiate the volume from. Note also that this uses the same Storage Class as the original volume, and also has the same size and access mode as the original volume.
% cat snapshot-restore.yaml apiVersion: v1 kind: PersistentVolumeClaim metadata: name: block-snapshot-restore spec: storageClassName: csisnaps-sc dataSource: name: block-snapshotvol kind: VolumeSnapshot apiGroup: snapshot.storage.k8s.io accessModes: - ReadWriteOnce resources: requests: storage: 2Gi
Let’s now apply this manifest, and check to see if we get a new volume combination for the restored snapshot.
% kubectl apply -f snapshot-restore.yaml persistentvolumeclaim/block-snapshot-restore created % kubectl get pvc,pv NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE persistentvolumeclaim/block-snapshot-restore Bound pvc-343fdfd7-e32d-4e0e-94c9-9a4e88c4999a 2Gi RWO csisnaps-sc 5s persistentvolumeclaim/csisnaps-pvc-vsan-claim Bound pvc-ca1960f2-6a86-43d6-80e4-6aab0f506e7a 2Gi RWO csisnaps-sc 30m NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE persistentvolume/pvc-343fdfd7-e32d-4e0e-94c9-9a4e88c4999a 2Gi RWO Delete Bound default/block-snapshot-restore csisnaps-sc 3s persistentvolume/pvc-ca1960f2-6a86-43d6-80e4-6aab0f506e7a 2Gi RWO Delete Bound default/csisnaps-pvc-vsan-claim csisnaps-sc 30m
Looks like it has worked. The final part is to create a pod which uses the newly restored snapshot, and verify that the data that we placed on the original volume was captured as part of the snapshot. Here is a new pod manifest which will mount the volume to the /snapshot folder.
% cat snapshot-restore-pod.yaml apiVersion: v1 kind: Pod metadata: name: csisnaps-restore-pod spec: containers: - name: busybox image: busybox volumeMounts: - name: csisnaps-snap mountPath: "/snapshot" command: [ "sleep", "1000000" ] volumes: - name: csisnaps-snap persistentVolumeClaim: claimName: block-snapshot-restore
Let’s create the pod and check the contents of the volume.
% kubectl apply -f snapshot-restore-pod.yaml pod/csisnaps-restore-pod created % kubectl get pods -w NAME READY STATUS RESTARTS AGE csisnaps-pod 1/1 Running 0 38m csisnaps-restore-pod 0/1 ContainerCreating 0 11s csisnaps-restore-pod 1/1 Running 0 12s ^C% % kubectl exec -it csisnaps-restore-pod -- sh / # df -h Filesystem Size Used Available Use% Mounted on overlay 77.7G 12.1G 61.6G 16% / tmpfs 64.0M 0 64.0M 0% /dev /dev/sdc 1.9G 6.0M 1.8G 0% /snapshot /dev/sda3 77.7G 12.1G 61.6G 16% /dev/termination-log /dev/sda3 77.7G 12.1G 61.6G 16% /etc/resolv.conf /dev/sda3 77.7G 12.1G 61.6G 16% /etc/hostname /dev/sda3 77.7G 12.1G 61.6G 16% /etc/hosts shm 64.0M 0 64.0M 0% /dev/shm tmpfs 15.5G 12.0K 15.5G 0% /var/run/secrets/kubernetes.io/serviceaccount tmpfs 7.8G 0 7.8G 0% /proc/acpi tmpfs 64.0M 0 64.0M 0% /proc/kcore tmpfs 64.0M 0 64.0M 0% /proc/keys tmpfs 64.0M 0 64.0M 0% /proc/timer_list tmpfs 7.8G 0 7.8G 0% /proc/scsi tmpfs 7.8G 0 7.8G 0% /sys/firmware / # ls /snapshot/ Special-Data lost+found / # cat /snapshot/Special-Data/special-data-file my-special-data
Looks good. We have successfully taken a snapshot of a volume, restore that volume to a PVC, and the resulting PV has our backed up data. While this might be somewhat useful to do on a manual basis, it will be extremely useful to our backup partners. There are a number of conversations already underway with these partners to allow their backup/restore products to integrate with vSphere CSI snapshots which will allow their products to backup/restore modern applications running on Kubernetes, which in turn is running on vSphere. Watch this space for more news about those initiatives.
The manifests demonstrated in this video are available for download on this GitHub repository.