Kubernetes Persistent Volume (PV) Encryption with Native Key Provider in vSphere 8.0U3

by Posted on

Security is top of mind for most, if not all, of our customers these days. Many years ago, I wrote a blog post on how customers could encrypt Kubernetes Persistent Volumes with an external Key Provider. One of our customers recently reached out to me to ask if we had any plans to provide similar support with the Native Key Provider. As my focus has been in other areas recently, I reached out to our CSI engineering team for an update. I then found out that support was added in our most recent release, vSphere 8.0U3. While no changes we necessary in the CSI driver, there were updates to our First Class Disks (FCD) mechanism which provides the backing for the Persistent Volumes. The result is that you can now encrypt Persistent Volumes using the vSphere CSI driver and the Native Key Provider in vSphere 8.0U3. Let’s take a closer look at how to do that next.

Versions

Probably important to show you the versions of the various components that I used to achieve this:

  • vCenter 8.0.3, build 24091160
  • VMware ESXi, 8.0.3, 24022510
  • Native Key Provider enabled on vSphere
  • Host Encryption Mode is enabled in the Security Profile of the ESXi hosts
  • Storage Policy using both vSAN and Encryption provided by Native KP
  • Ubuntu OS version 24.04
  • Upstream K8s version 1.30.3
  • vSphere Container Storage Plugin / CSI driver version v3.0.0
  • Works with and without vTPM devices on K8s node/VM, so vTPM is optional

Encryption Storage Policy

I built the storage policy using a combination of vSAN rules and Encryption rules. Here is the policy:

Encrypted Kubernetes Nodes

Next, I created a simple Kubernetes cluster, stood up the hard way with kubeadm. I used the storage policy with encryption enabled when deploying the VM. This would become the K8s node. I also ensured that the disk.EnableUUID was set to true on all nodes, and swap was disabled. I installed the vSphere Container Storage Plugin using the official docs. Once the cluster was deployed, I checked the node/VM encryption state in the vSphere Client. Everything appeared to be as expected, and the node/VM was encrypted using the Native KP:

Create and use Encrypted Persistent Volume

Now it was time to test whether or not I could create some encrypted Persistent Volumes on this K8s cluster. What follows are 3 YAML manifests, one for Storage Class (which uses encryption), one for Persistent Volume Claim (PVC) and a final one for a Pod which will mount the volume.

Storage Class

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: vsan-encrypt-sc
provisioner: csi.vsphere.vmware.com
parameters:
  storagepolicyname: "vSAN+VMCrypt"

PVC

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: encrypt-pvc
spec:
  storageClassName: vsan-encrypt-sc
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 3Gi

Pod

apiVersion: v1
kind: Pod
metadata:
  name: encrypt-pod-a
spec:
  containers:
  - name: encrypt-pod-a
    image: "k8s.gcr.io/busybox"
    volumeMounts:
    - name: encrypt-vol
      mountPath: "/mnt/volume1"
    command: [ "sleep", "1000000" ]
  volumes:
    - name: encrypt-vol
      persistentVolumeClaim:
        claimName: encrypt-pvc

Once deployed, the following objects are created, including the Persistent Volume (which of course should now be encrypted):

$ kubectl get pod,pvc,pv,sc
NAME                READY   STATUS    RESTARTS   AGE
pod/encrypt-pod-a   1/1     Running   0          16h

NAME                                STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS      VOLUMEATTRIBUTESCLASS   AGE
persistentvolumeclaim/encrypt-pvc   Bound    pvc-c391df7d-1e6c-4488-b44a-a96917540d6f   3Gi        RWO            vsan-encrypt-sc   <unset>                 18h

NAME                                                        CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                 STORAGECLASS      VOLUMEATTRIBUTESCLASS   REASON   AGE
persistentvolume/pvc-c391df7d-1e6c-4488-b44a-a96917540d6f   3Gi        RWO            Delete           Bound    default/encrypt-pvc   vsan-encrypt-sc   <unset>                          17h

NAME                                          PROVISIONER              RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
storageclass.storage.k8s.io/vsan-encrypt-sc   csi.vsphere.vmware.com   Delete          Immediate           false                  20h

And if we check the VM/node settings back on the vSphere Client and examine the Hard Disk which represents the Persistent Volume, we do indeed see that it is encrypted:

Summary

This is a great new feature in vSphere 8.0U3. Customers can now use the embedded vSphere Native Key Provider to encrypt Kubernetes nodes and Persistent Volumes. I am certain that this is a feature that many of our customers will be keen to leverage for their Kubernetes on vSphere deployments.

Published by Cormac