Open Source Velero Plugin for vSphere (using snapshots) in action

I recently published an article around Velero and vSAN File Services, showing how Velero and the restic plugin could be used to backup and restore Kubernetes application that used vSAN File Services. Today, I want to turn my attention to a very cool new plugin that we announced in mid-April, namely the Velero Plugin for vSphere. This open source plugin enables Velero to take a crash-consistent VADP* snapshot backup of a block Persistent Volume on vSphere storage, and store the backup on S3 compatible storage.

* VADP is short for VMware vSphere Storage APIs – Data Protection.

To utilize the plugin, I deployed the evaluation Minio S3 object store which comes with Velero, then I went ahead with the install of Velero on vSphere as per normal (which on vSphere means you install it with the AWS plugin). This is because the vSphere plugin does not provide object storage, so we are leveraging the AWS plugin and the evaluation Minio S3 object store. You can also include the restic plugin to give you some different backup options. Once Velero is installed, you simply need to install the Velero Plugin for vSphere, then create a Volume Snapshot Location (VSL). Now you are ready to use Velero Plugin for vSphere to backup and restore applications using PVs on vSphere storage.

Let’s look at the steps in more detail. Note that this is Velero version 1.3.2. The latest version of Velero, v1.4, released in late May 2020. I am not going to detail how to install the evaluation Minio S3 or how to install Velero. This is well documented in various places. I will start with the installation of the Velero Plugin for vSphere. This is very simple, as you can see in step 1 below.

Step 1 – Install the Velero Plugin for vSphere

$ velero plugin get
NAME                              KIND
velero.io/crd-remap-version       BackupItemAction
velero.io/pod                     BackupItemAction
velero.io/pv                      BackupItemAction
velero.io/service-account         BackupItemAction
velero.io/aws                     ObjectStore
velero.io/add-pv-from-pvc         RestoreItemAction
velero.io/add-pvc-from-pod        RestoreItemAction
velero.io/change-storage-class    RestoreItemAction
velero.io/cluster-role-bindings   RestoreItemAction
velero.io/crd-preserve-fields     RestoreItemAction
velero.io/job                     RestoreItemAction
velero.io/pod                     RestoreItemAction
velero.io/restic                  RestoreItemAction
velero.io/role-bindings           RestoreItemAction
velero.io/service                 RestoreItemAction
velero.io/service-account         RestoreItemAction
velero.io/aws                     VolumeSnapshotter


$ velero plugin add vsphereveleroplugin/velero-plugin-for-vsphere:1.0.0


$ velero plugin get
NAME                              KIND
velero.io/crd-remap-version       BackupItemAction
velero.io/pod                     BackupItemAction
velero.io/pv                      BackupItemAction
velero.io/service-account         BackupItemAction
velero.io/aws                     ObjectStore
velero.io/add-pv-from-pvc         RestoreItemAction
velero.io/add-pvc-from-pod        RestoreItemAction
velero.io/change-storage-class    RestoreItemAction
velero.io/cluster-role-bindings   RestoreItemAction
velero.io/crd-preserve-fields     RestoreItemAction
velero.io/job                     RestoreItemAction
velero.io/pod                     RestoreItemAction
velero.io/restic                  RestoreItemAction
velero.io/role-bindings           RestoreItemAction
velero.io/service                 RestoreItemAction
velero.io/service-account         RestoreItemAction
velero.io/aws                     VolumeSnapshotter
velero.io/vsphere                 VolumeSnapshotter

Step 2 – Create a  Volume Snapshot Location

Because Velero is deployed on vSphere, it is configured with the AWS plugin. Thus backups are stored in an S3 bucket. In this demo, I mentioned that I also deployed the evaluation Minio S3 object store. Volume Snapshot backups taken by the Velero Plugin for vSphere are stored in the same S3 bucket as the application metadata on the Minio S3 object store.

The next step is to create a  Volume Snapshot Location. This  Volume Snapshot Location is referenced when a backup is taken. The Volume Snapshot Location is used to specify the use of the Velero Plugin for vSphere.

$ velero snapshot-location create vsl-vsphere --provider velero.io/vsphere
Snapshot volume location "vsl-vsphere" configured successfully.

Step 3 – Backup using the Plugin and Volume Snapshot Location

Time to take a backup. In this test, I am backing up a complete namespace. This namespace holds a Cassandra StatefulSet which has 3 replicas. A backup will include Kubernetes objects such as Pods, Services, PVCs and PVs. The Cassandra application has a few simple populated tables. Note how the velero backup command references the VSL created in step 2. This tells the backup to use the vSphere plugin.

$ velero backup create cassandra-snap-backup --include-namespaces=cassandra \
--snapshot-volumes --volume-snapshot-locations vsl-vsphere
Backup request "cassandra-snap-backup" submitted successfully.
Run `velero backup describe cassandra-snap-backup` or `velero backup logs cassandra-snap-backup` for more details.


$ velero backup describe cassandra-snap-backup
Name:         cassandra-snap-backup
Namespace:    velero
Labels:       velero.io/storage-location=default
Annotations:  <none>

Phase:  InProgress

Namespaces:
  Included:  cassandra
  Excluded:  <none>

Resources:
  Included:        *
  Excluded:        <none>
  Cluster-scoped:  auto

Label selector:  <none>

Storage Location:  default

Snapshot PVs:  true

TTL:  720h0m0s

Hooks:  <none>

Backup Format Version:  1

Started:    2020-05-26 11:10:58 +0100 IST
Completed:  <n/a>

Expiration:  2020-06-25 11:10:58 +0100 IST

Persistent Volumes: <none included>

Some time later …

$ velero backup get
NAME                    STATUS      CREATED                         EXPIRES   STORAGE LOCATION   SELECTOR
cassandra-snap-backup   Completed   2020-05-26 11:10:58 +0100 IST   29d       default            <none>


$ velero backup describe cassandra-snap-backup --details
Name:         cassandra-snap-backup
Namespace:    velero
Labels:       velero.io/storage-location=default
Annotations:  <none>

Phase:  Completed

Namespaces:
  Included:  cassandra
  Excluded:  <none>

Resources:
  Included:        *
  Excluded:        <none>
  Cluster-scoped:  auto

Label selector:  <none>

Storage Location:  default

Snapshot PVs:  true

TTL:  720h0m0s

Hooks:  <none>

Backup Format Version:  1

Started:    2020-05-26 11:10:58 +0100 IST
Completed:  2020-05-26 11:11:32 +0100 IST

Expiration:  2020-06-25 11:10:58 +0100 IST

Resource List:
  apps/v1/ControllerRevision:
    - cassandra/cassandra-dd67c56d6
  apps/v1/StatefulSet:
    - cassandra/cassandra
  v1/Endpoints:
    - cassandra/cassandra
  v1/Namespace:
    - cassandra
  v1/PersistentVolume:
    - pvc-5802b03d-f875-4cf6-9a8c-7626856cdeef
    - pvc-98914696-db27-49f9-807b-f2d29c25cc65
    - pvc-dce81a7e-4758-4c4b-ba26-eff658e93be0
  v1/PersistentVolumeClaim:
    - cassandra/cassandra-data-cassandra-0
    - cassandra/cassandra-data-cassandra-1
    - cassandra/cassandra-data-cassandra-2
  v1/Pod:
    - cassandra/cassandra-0
    - cassandra/cassandra-1
    - cassandra/cassandra-2
  v1/Secret:
    - cassandra/default-token-ghmwc
  v1/Service:
    - cassandra/cassandra
  v1/ServiceAccount:
    - cassandra/default

Persistent Volumes:
  pvc-5802b03d-f875-4cf6-9a8c-7626856cdeef:
    Snapshot ID:        ivd:bc611f64-f318-46c6-bcdf-046d6e0f50b6:ef06b313-4593-4fca-bbca-4dd319a34e86
    Type:               ivd
    Availability Zone:
    IOPS:               <N/A>
  pvc-dce81a7e-4758-4c4b-ba26-eff658e93be0:
    Snapshot ID:        ivd:419c9163-87d0-4c59-be02-1407f2fc3ae3:a40e159c-dffe-4197-b1f5-e9e7cd7c48cc
    Type:               ivd
    Availability Zone:
    IOPS:               <N/A>
  pvc-98914696-db27-49f9-807b-f2d29c25cc65:
    Snapshot ID:        ivd:8dc53f9c-523a-43c6-b3b8-9bc3a0f3c893:4d42acdb-0e73-40ae-9c62-812e995829a5
    Type:               ivd
    Availability Zone:

At the end of the detailed output above, we can see 3 snapshots representing the backups of the 3 PVs. The IVD references is shorthand for Improved Virtual Disks, another name for First Class Disks or FCDs. More details on IVDs/FCDs can be found here and here.

During the backup, you will observe a number of events occurring in vSphere. You will seen a ‘Create virtual object snapshot’ event when the backup is initiated.

And there is a corresponding Delete virtual object snapshot’ event when the backup is completed.

Important: The data movement task, which we refer to as an upload, copies the local on-disk snapshot to the S3 object store. This operation is asynchronous to the Velero backup. Thus when the velero backup displays a Completed status, this does not mean the the upload task is completed. Administrators should use the kubectl -n velero get uploads command to monitor the upload tasks. Do not initiate any velero restore commands on the backup until all relevant upload tasks for the backup are completed.

Here is an example of such a command, along with a describe. There will be an upload per snapshot, so for my Cassandra demo, there are 3 uploads per backup. The describe option can be used to display more information about the upload.

$ kubectl -n velero get uploads
NAME                                          AGE
upload-4d42acdb-0e73-40ae-9c62-812e995829a5   22h
upload-a40e159c-dffe-4197-b1f5-e9e7cd7c48cc   22h
upload-ef06b313-4593-4fca-bbca-4dd319a34e86   22h


$ kubectl -n velero describe uploads upload-ef06b313-4593-4fca-bbca-4dd319a34e86
Name:         upload-ef06b313-4593-4fca-bbca-4dd319a34e86
Namespace:    velero
Labels:       <none>
Annotations:  <none>
API Version:  veleroplugin.io/v1
Kind:         Upload
Metadata:
  Creation Timestamp:  2020-05-26T10:11:17Z
  Generation:          3
  Resource Version:    6729495
  Self Link:           /apis/veleroplugin.io/v1/namespaces/velero/uploads/upload-ef06b313-4593-4fca-bbca-4dd319a34e86
  UID:                 39608852-903e-468d-975e-397f51ebb3db
Spec:
  Backup Timestamp:  2020-05-26T10:11:18Z
  Snapshot ID:       ivd:bc611f64-f318-46c6-bcdf-046d6e0f50b6:ef06b313-4593-4fca-bbca-4dd319a34e86
Status:
  Completion Timestamp:  2020-05-26T10:16:19Z
  Message:               Upload completed
  Next Retry Timestamp:  2020-05-26T10:11:18Z
  Phase:                 Completed
  Processing Node:       k8s2-worker-02
  Progress:
  Start Timestamp:  2020-05-26T10:11:19Z
Events:             <none>

Step 4 – Check the backups in the S3 object store

Data is stored on the S3 object store in a different way to the contents of a restic backup. There is now a separate plugin folder in the velero bucket which contains IVD snapshot data, as shown below.

Step 5 – Initiate Restore

A velero restore using the vSphere plugin is exactly the same as many restores that we have looked at before on this blog. Again, I can do something catastrophic like remove the Cassandra namespace and then try to restore it.

$ kubectl delete ns cassandra
namespace "cassandra" deleted

$ kubectl get ns
NAME              STATUS   AGE
default           Active   27d
kube-node-lease   Active   27d
kube-public       Active   27d
kube-system       Active   27d
velero            Active   3d23h
vsan-prometheus   Active   18d


$ velero restore create cassandra-restore --from-backup cassandra-snap-backup
Restore request "cassandra-restore" submitted successfully.
Run `velero restore describe cassandra-restore` or `velero restore logs cassandra-restore` for more details.


$ velero restore describe cassandra-restore
Name:         cassandra-restore
Namespace:    velero
Labels:       <none>
Annotations:  <none>

Phase:  InProgress

Backup:  cassandra-snap-backup

Namespaces:
  Included:  *
  Excluded:  <none>

Resources:
  Included:        *
  Excluded:        nodes, events, events.events.k8s.io, backups.velero.io, restores.velero.io, resticrepositories.velero.io
  Cluster-scoped:  auto

Namespace mappings:  <none>

Label selector:  <none>

Restore PVs:  auto

Some time later:

$ velero restore describe cassandra-restore --details
Name:         cassandra-restore
Namespace:    velero
Labels:       <none>
Annotations:  <none>

Phase:  Completed

Backup:  cassandra-snap-backup

Namespaces:
  Included:  *
  Excluded:  <none>

Resources:
  Included:        *
  Excluded:        nodes, events, events.events.k8s.io, backups.velero.io, restores.velero.io, resticrepositories.velero.io
  Cluster-scoped:  auto

Namespace mappings:  <none>

Label selector:  <none>

Restore PVs:  auto

Important: As mentioned previously, velero backup completion does not necessarily mean the backup of the snapshot data has been completed. However velero restore completion does indeed mean the restore of the snapshot data has been completed, so slightly different to backup behaviour. Restoring the snapshot from the S3 object store is referred to as a download, and just like we did with backup, we can use the kubectl -n velero get downloads command to monitor those tasks

$ kubectl -n velero get downloads
NAME                                                                               AGE
download-4d42acdb-0e73-40ae-9c62-812e995829a5-40769481-a685-4ad8-8ce5-5a0a3ff121f4 21h
download-a40e159c-dffe-4197-b1f5-e9e7cd7c48cc-9ad8a9cb-3043-433c-8111-13d923fc0df2 21h
download-ef06b313-4593-4fca-bbca-4dd319a34e86-863f2aa1-7346-4093-adb4-6191b3c2d03a 21h

$ kubectl -n velero describe downloads download-ef06b313-4593-4fca-bbca-4dd319a34e86-863f2aa1-7346-4093-adb4-6191b3c2d03a
Name:         download-ef06b313-4593-4fca-bbca-4dd319a34e86-863f2aa1-7346-4093-adb4-6191b3c2d03a
Namespace:    velero
Labels:       <none>
Annotations:  <none>
API Version:  veleroplugin.io/v1
Kind:         Download
Metadata:
  Creation Timestamp:  2020-05-26T11:23:11Z
  Generation:          3
  Resource Version:    6742468
  Self Link:           /apis/veleroplugin.io/v1/namespaces/velero/downloads/download-ef06b313-4593-4fca-bbca-4dd319a34e86-863f2aa1-7346-4093-adb4-6191b3c2d03a
  UID:                 9d2164e6-210d-4058-a6b5-de477b164cb0
Spec:
  Restore Timestamp:  2020-05-26T11:23:12Z
  Snapshot ID:        ivd:bc611f64-f318-46c6-bcdf-046d6e0f50b6:ef06b313-4593-4fca-bbca-4dd319a34e86
Status:
  Completion Timestamp:  2020-05-26T11:27:50Z
  Message:               Download completed
  Next Retry Timestamp:  2020-05-26T11:23:12Z
  Phase:                 Completed
  Processing Node:       k8s2-worker-03
  Progress:
  Start Timestamp:  2020-05-26T11:23:13Z
  Volume ID:        ivd:dbf0a76a-4634-49cf-a645-4faac2c989f0
Events:             <none>

Once the backup has completed and the downloads have completed, we can check on the application.

$ kubectl get sts -n cassandra
NAME        READY   AGE
cassandra   3/3     6m4s


$ kubectl get pvc -n cassandra
NAME                         STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
cassandra-data-cassandra-0   Bound    pvc-5802b03d-f875-4cf6-9a8c-7626856cdeef   1Gi        RWO            cass-sc-csi    6m10s
cassandra-data-cassandra-1   Bound    pvc-dce81a7e-4758-4c4b-ba26-eff658e93be0   1Gi        RWO            cass-sc-csi    6m10s
cassandra-data-cassandra-2   Bound    pvc-98914696-db27-49f9-807b-f2d29c25cc65   1Gi        RWO            cass-sc-csi    6m10s


$ kubectl get pod -n cassandra
NAME          READY   STATUS    RESTARTS   AGE
cassandra-0   1/1     Running   0          6m14s
cassandra-1   1/1     Running   3          6m14s
cassandra-2   1/1     Running   3          6m14s

Conclusion

Success! We have backed up and restored a Cassandra application using the Velero plugin for vSphere, which in turn leverages VADP crash-consistent snapshots rather than restic file copies. The Velero plugin for vSphere is now supported with Velero 1.4.

The team would be delighted if you could try this out and provide some feedback. An FAQ with known issues is available. If you encounter issues, please add them here for review by the team.

One final note: I was able to do backups with both the vSphere plugin and restic plugin. So both can co-exist, and both can be called to backup different Kubernetes resources if necessary.

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.