Site icon CormacHogan.com

Announcing vSphere CSI driver v2.5 support for CSI snapshots

I’m delighted to announce that VMware has just released a new version of the vSphere CSI driver for Kubernetes. The full release notes are available here. This 2.5 version now has support for CSI snapshots, a feature that many customers and partners have been waiting for. This features enables snapshots to be taken of block-based persistent volumes, as well as the ability to restore snapshots to persistent volumes. Those of you familiar with the CSI driver will notice two additional Kubernetes components to enable snapshot support. The first is a new snapshot controller. When a snapshot is created, two new objects are created;  a VolumeSnapshot and a VolumeSnapshotContent. Simply put, one can consider a VolumeSnapshot to being something akin a to PersistentVolumeClaim and a VolumeSnapshotContent object as being something akin to a PersistentVolume. The snapshot controller is responsible for the dynamic creation and  deletion of snapshots, and for binding VolumeSnapshot requests with its VolumeSnapshotContents in the backed. The second component is a new sidecar that has been added to the CSI controller pod called csi-snapshotter. This triggers the CreateSnapshot and DeleteSnapshot operations.

Requirements, steps to enable Volume Snapshot and Restore as well as configuration parameters can be found here. The official docs include a link to a bash script which checks that the snapshot feature has been enabled, pulls the necessary manifest to install the necessary snapshot components, deploys the snapshot controller, and finally updates the CSI driver with the additional snapshot sidecar. The script is available here. Here is a sample output from my Kubernetes cluster when running the deployment bash script.

% bash deploy-csi-snapshot-components.sh
No existing snapshot-controller Deployment found, deploying it now..
Start snapshot-controller deployment...
customresourcedefinition.apiextensions.k8s.io/volumesnapshotclasses.snapshot.storage.k8s.io created
Created CRD volumesnapshotclasses.snapshot.storage.k8s.io
customresourcedefinition.apiextensions.k8s.io/volumesnapshotcontents.snapshot.storage.k8s.io created
Created CRD volumesnapshotcontents.snapshot.storage.k8s.io
customresourcedefinition.apiextensions.k8s.io/volumesnapshots.snapshot.storage.k8s.io created
Created CRD volumesnapshots.snapshot.storage.k8s.io
✅ Deployed VolumeSnapshot CRDs
serviceaccount/snapshot-controller unchanged
clusterrole.rbac.authorization.k8s.io/snapshot-controller-runner unchanged
clusterrolebinding.rbac.authorization.k8s.io/snapshot-controller-role unchanged
role.rbac.authorization.k8s.io/snapshot-controller-leaderelection unchanged
rolebinding.rbac.authorization.k8s.io/snapshot-controller-leaderelection unchanged
✅ Created  RBACs for snapshot-controller
deployment.apps/snapshot-controller created
deployment.apps/snapshot-controller image updated
deployment.apps/snapshot-controller patched
deployment.apps/snapshot-controller patched
Waiting for deployment spec update to be observed...
Waiting for deployment "snapshot-controller" rollout to finish: 0 out of 2 new replicas have been updated...
Waiting for deployment "snapshot-controller" rollout to finish: 1 out of 2 new replicas have been updated...
Waiting for deployment "snapshot-controller" rollout to finish: 1 out of 2 new replicas have been updated...
Waiting for deployment "snapshot-controller" rollout to finish: 1 out of 2 new replicas have been updated...
Waiting for deployment "snapshot-controller" rollout to finish: 1 out of 2 new replicas have been updated...
Waiting for deployment "snapshot-controller" rollout to finish: 1 out of 2 new replicas have been updated...
Waiting for deployment "snapshot-controller" rollout to finish: 1 out of 2 new replicas have been updated...
Waiting for deployment "snapshot-controller" rollout to finish: 1 out of 2 new replicas have been updated...
Waiting for deployment "snapshot-controller" rollout to finish: 1 of 2 updated replicas are available...
Waiting for deployment "snapshot-controller" rollout to finish: 1 of 2 updated replicas are available...
deployment "snapshot-controller" successfully rolled out

✅ Successfully deployed snapshot-controller

No existing snapshot-validation-deployment Deployment found, deploying it now..
creating certs in tmpdir /var/folders/31/y77ywvzd6lqc0g60r4xnfyd80000gp/T/tmp.HmdOwrGg7f 
Generating a 2048 bit RSA private key
.............................................................................................+++
.........................+++
writing new private key to '/var/folders/31/y77ywvzd6lqc0g60r4xnfyd80000gp/T/tmp.HmdOwrGg7f/ca.key'
-----
Generating RSA private key, 2048 bit long modulus
...............................................................+++
.............................................................................+++
e is 65537 (0x10001)
Signature ok
subject=/CN=snapshot-validation-service.kube-system.svc
Getting CA Private Key
secret "snapshot-webhook-certs" deleted
secret/snapshot-webhook-certs created
service "snapshot-validation-service" deleted
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  2238  100  2238    0     0   9060      0 --:--:-- --:--:-- --:--:--  9024
service/snapshot-validation-service created
validatingwebhookconfiguration.admissionregistration.k8s.io/validation-webhook.snapshot.storage.k8s.io configured
deployment.apps/snapshot-validation-deployment created
deployment.apps/snapshot-validation-deployment image updated
deployment.apps/snapshot-validation-deployment patched
Waiting for deployment spec update to be observed...
Waiting for deployment spec update to be observed...
Waiting for deployment "snapshot-validation-deployment" rollout to finish: 0 out of 3 new replicas have been updated...
Waiting for deployment "snapshot-validation-deployment" rollout to finish: 0 out of 3 new replicas have been updated...
Waiting for deployment "snapshot-validation-deployment" rollout to finish: 1 out of 3 new replicas have been updated...
Waiting for deployment "snapshot-validation-deployment" rollout to finish: 1 out of 3 new replicas have been updated...
Waiting for deployment "snapshot-validation-deployment" rollout to finish: 1 out of 3 new replicas have been updated...
Waiting for deployment "snapshot-validation-deployment" rollout to finish: 1 out of 3 new replicas have been updated...
Waiting for deployment "snapshot-validation-deployment" rollout to finish: 1 out of 3 new replicas have been updated...
Waiting for deployment "snapshot-validation-deployment" rollout to finish: 2 out of 3 new replicas have been updated...
Waiting for deployment "snapshot-validation-deployment" rollout to finish: 2 out of 3 new replicas have been updated...
Waiting for deployment "snapshot-validation-deployment" rollout to finish: 2 out of 3 new replicas have been updated...
Waiting for deployment "snapshot-validation-deployment" rollout to finish: 2 out of 3 new replicas have been updated...
Waiting for deployment "snapshot-validation-deployment" rollout to finish: 1 old replicas are pending termination...
Waiting for deployment "snapshot-validation-deployment" rollout to finish: 1 old replicas are pending termination...
deployment "snapshot-validation-deployment" successfully rolled out

✅ Successfully deployed snapshot-validation-deployment

csi-snapshotter side-car not found in vSphere CSI Driver Deployment, patching..
creating patch file in tmpdir /var/folders/31/y77ywvzd6lqc0g60r4xnfyd80000gp/T/tmp.GiMDimfZdq
Scale down the vSphere CSI driver
deployment.apps/vsphere-csi-controller scaled
Patching vSphere CSI driver..
deployment.apps/vsphere-csi-controller patched
Scaling the vSphere CSI driver back to original state..
deployment.apps/vsphere-csi-controller scaled
Waiting for deployment spec update to be observed...
Waiting for deployment spec update to be observed...
Waiting for deployment "vsphere-csi-controller" rollout to finish: 0 out of 3 new replicas have been updated...
Waiting for deployment "vsphere-csi-controller" rollout to finish: 0 of 3 updated replicas are available...
Waiting for deployment "vsphere-csi-controller" rollout to finish: 1 of 3 updated replicas are available...
Waiting for deployment "vsphere-csi-controller" rollout to finish: 2 of 3 updated replicas are available...
deployment "vsphere-csi-controller" successfully rolled out

✅ Successfully deployed all components for CSI Snapshot feature!

If the update has been successful, you should observe 7 containers in the vSphere CSI controller pod.

% kubectl get pod -n vmware-system-csi vsphere-csi-controller-5b6dfc6799-lmphc
NAME                                      READY  STATUS    RESTARTS      AGE
vsphere-csi-controller-5b6dfc6799-lmphc   7/7    Running   7 (12d ago)   20d

In this post, I will take an existing StorageClass, PVC, PV and pod deployment. I will then snapshot the volume, convert the snapshot to a new volume, and then mount the snapshot’ed volume to another pod. Lastly, I will verify that the data on the volume is present and available.

Here is the current test app deployment. I have a pod, PVC, PV and StorageClass already built. The pod is a very simple busybox pod which has some basic Unix utilities incorporated.

% kubectl get sc,pod,pvc,pv
NAME                                                PROVISIONER              RECLAIMPOLICY  VOLUMEBINDINGMODE  ALLOWVOLUMEEXPANSION  AGE
storageclass.storage.k8s.io/csisnaps-sc (default)  csi.vsphere.vmware.com  Delete          Immediate          true                  10s


NAME              READY  STATUS    RESTARTS  AGE
pod/csisnaps-pod  1/1    Running  0          9s


NAME                                            STATUS  VOLUME                                    CAPACITY  ACCESS MODES  STORAGECLASS  AGE
persistentvolumeclaim/csisnaps-pvc-vsan-claim  Bound    pvc-ca1960f2-6a86-43d6-80e4-6aab0f506e7a  2Gi        RWO            csisnaps-sc    9s


NAME                                                        CAPACITY  ACCESS MODES  RECLAIM POLICY  STATUS  CLAIM                            STORAGECLASS  REASON  AGE
persistentvolume/pvc-ca1960f2-6a86-43d6-80e4-6aab0f506e7a  2Gi        RWO            Delete          Bound    default/csisnaps-pvc-vsan-claim  csisnaps-sc            9s

Let’s add some simple data to the volume mounted to the busybox pod by ‘exec’ing into it and creating a directory and a file with some contents.

% kubectl exec -it csisnaps-pod -- sh
/ # df -h
Filesystem                Size     Used  Available  Use% Mounted on
overlay                  77.7G    12.1G     61.6G   16%  /
tmpfs                    64.0M        0     64.0M    0%  /dev
/dev/sdb                  1.9G     6.0M      1.8G    0%  /demo
/dev/sda3                77.7G    12.1G     61.6G   16%  /dev/termination-log
/dev/sda3                77.7G    12.1G     61.6G   16%  /etc/resolv.conf
/dev/sda3                77.7G    12.1G     61.6G   16%  /etc/hostname
/dev/sda3                77.7G    12.1G     61.6G   16%  /etc/hosts
shm                      64.0M        0     64.0M   0%   /dev/shm
tmpfs                    15.5G    12.0K     15.5G   0%   /var/run/secrets/kubernetes.io/serviceaccount
tmpfs                    7.8G         0      7.8G   0%   /proc/acpi
tmpfs                    64.0M        0     64.0M   0%   /proc/kcore
tmpfs                    64.0M        0     64.0M   0%   /proc/keys
tmpfs                    64.0M        0     64.0M   0%   /proc/timer_list
tmpfs                    7.8G         0      7.8G   0%   /proc/scsi
tmpfs                    7.8G         0      7.8G   0%   /sys/firmware
/ # cd /demo
/demo # ls
lost+found
/demo # mkdir Special-Data
/demo # cd Special-Data/
/demo/Special-Data # echo "my-special-data" >> special-data-file
/demo/Special-Data # cat special-data-file
my-special-data
/demo/Special-Data #

Now we will take a snapshot of this volume. These are the manifest that I will use to first make a CSI snapshot class which defines which provider should be used to create snapshots, and  what to do when a snapshot is deleted. The second manifest is used to make a snapshot. Note that the snapshot manifest contains a spec.source.persistentVolumeClaimName which determines which volume should be snapshot’ed. In this examples, it matches the PVC called csisnaps-pvc-vsan-claim, as shown above.

% cat csi-snapshot-class.yaml
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshotClass
metadata:
  name: block-snapshotclass
driver: csi.vsphere.vmware.com
deletionPolicy: Delete


% cat dynamic-vol-snap.yaml
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
  name: block-snapshotvol
spec:
  volumeSnapshotClassName: block-snapshotclass
  source:
    persistentVolumeClaimName: csisnaps-pvc-vsan-claim

Let’s now apply the manifests, and then check the VolumeSnapshot and the VolumeSnapshotContents.

% kubectl apply -f dynamic-vol-snap.yaml
volumesnapshot.snapshot.storage.k8s.io/block-snapshotvol created


% kubectl get volumesnapshot
NAME                READYTOUSE   SOURCEPVC                 SOURCESNAPSHOTCONTENT   RESTORESIZE   SNAPSHOTCLASS         SNAPSHOTCONTENT                                    CREATIONTIME   AGE
block-snapshotvol   true         csisnaps-pvc-vsan-claim                           2Gi           block-snapshotclass   snapcontent-2471e409-7e7e-4fed-8d7a-95a73b097abf   24s            30s


% kubectl get volumesnapshotcontents
NAME                                               READYTOUSE   RESTORESIZE   DELETIONPOLICY   DRIVER                   VOLUMESNAPSHOTCLASS   VOLUMESNAPSHOT      VOLUMESNAPSHOTNAMESPACE   AGE
snapcontent-2471e409-7e7e-4fed-8d7a-95a73b097abf   true         2147483648    Delete           csi.vsphere.vmware.com   block-snapshotclass   block-snapshotvol   default                   34s

Great! It looks like a snapshot of the volume used by my simple application has been taken successfully. The next step is to restore that snapshot which will instantiate a new volume. Once that is successful, the volume can then be mounted to a pod where we can check its contents to ensure the volume data was indeed captured by the snapshot. First, I have to construct a snapshot restore manifest which is basically a PVC that includes a spec.datasource.kind set to VolumeSnapshot, along with the name of the snapshot we wish to instantiate the volume from. Note also that this uses the same Storage Class as the original volume, and also has the same size and access mode as the original volume.

% cat snapshot-restore.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: block-snapshot-restore
spec:
  storageClassName: csisnaps-sc
  dataSource:
    name: block-snapshotvol
    kind: VolumeSnapshot
    apiGroup: snapshot.storage.k8s.io
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 2Gi

Let’s now apply this manifest, and check to see if we get a new volume combination for the restored snapshot.

% kubectl apply -f snapshot-restore.yaml
persistentvolumeclaim/block-snapshot-restore created


% kubectl get pvc,pv
NAME                                            STATUS  VOLUME                                    CAPACITY  ACCESS MODES    STORAGECLASS   AGE
persistentvolumeclaim/block-snapshot-restore    Bound    pvc-343fdfd7-e32d-4e0e-94c9-9a4e88c4999a  2Gi        RWO            csisnaps-sc    5s
persistentvolumeclaim/csisnaps-pvc-vsan-claim   Bound    pvc-ca1960f2-6a86-43d6-80e4-6aab0f506e7a  2Gi        RWO            csisnaps-sc    30m

NAME                                                        CAPACITY  ACCESS MODES  RECLAIM POLICY  STATUS  CLAIM                              STORAGECLASS   REASON  AGE
persistentvolume/pvc-343fdfd7-e32d-4e0e-94c9-9a4e88c4999a  2Gi        RWO            Delete          Bound    default/block-snapshot-restore    csisnaps-sc            3s
persistentvolume/pvc-ca1960f2-6a86-43d6-80e4-6aab0f506e7a  2Gi        RWO            Delete          Bound    default/csisnaps-pvc-vsan-claim   csisnaps-sc            30m

Looks like it has worked. The final part is to create a pod which uses the newly restored snapshot, and verify that the data that we placed on the original volume was captured as part of the snapshot. Here is a new pod manifest which will mount the volume to the /snapshot folder.

% cat snapshot-restore-pod.yaml
apiVersion: v1
kind: Pod
metadata:
  name: csisnaps-restore-pod
spec:
  containers:
  - name: busybox
    image: busybox
    volumeMounts:
    - name: csisnaps-snap
      mountPath: "/snapshot"
    command: [ "sleep", "1000000" ]
  volumes:
    - name: csisnaps-snap
      persistentVolumeClaim:
        claimName: block-snapshot-restore

Let’s create the pod and check the contents of the volume.

% kubectl apply -f snapshot-restore-pod.yaml
pod/csisnaps-restore-pod created


% kubectl get pods -w
NAME                  READY  STATUS              RESTARTS  AGE
csisnaps-pod          1/1    Running            0          38m
csisnaps-restore-pod  0/1    ContainerCreating  0          11s
csisnaps-restore-pod  1/1    Running            0          12s
^C%


% kubectl exec -it csisnaps-restore-pod -- sh
/ # df -h
Filesystem                Size      Used Available Use% Mounted on
overlay                  77.7G     12.1G    61.6G   16% /
tmpfs                    64.0M         0    64.0M    0% /dev
/dev/sdc                  1.9G      6.0M     1.8G    0% /snapshot
/dev/sda3                77.7G     12.1G    61.6G   16% /dev/termination-log
/dev/sda3                77.7G     12.1G    61.6G   16% /etc/resolv.conf
/dev/sda3                77.7G     12.1G    61.6G   16% /etc/hostname
/dev/sda3                77.7G     12.1G    61.6G   16% /etc/hosts
shm                      64.0M         0    64.0M    0% /dev/shm
tmpfs                    15.5G     12.0K    15.5G    0% /var/run/secrets/kubernetes.io/serviceaccount
tmpfs                    7.8G          0     7.8G    0% /proc/acpi
tmpfs                    64.0M         0    64.0M    0% /proc/kcore
tmpfs                    64.0M         0    64.0M    0% /proc/keys
tmpfs                    64.0M         0    64.0M    0% /proc/timer_list
tmpfs                    7.8G          0     7.8G    0% /proc/scsi
tmpfs                    7.8G          0     7.8G    0% /sys/firmware
/ # ls /snapshot/
Special-Data  lost+found
/ # cat /snapshot/Special-Data/special-data-file
my-special-data

Looks good. We have successfully taken a snapshot of a volume, restore that volume to a PVC, and the resulting PV has our backed up data. While this might be somewhat useful to do on a manual basis, it will be extremely useful to our backup partners. There are a number of conversations already underway with these partners to allow their backup/restore products to integrate with vSphere CSI snapshots which will allow their products to backup/restore modern applications running on Kubernetes, which in turn is running on vSphere. Watch this space for more news about those initiatives.

The manifests demonstrated in this video are available for download on this GitHub repository.

Exit mobile version