You might ask why you would want a static volume to be provisioned in the first place? This is eloquently answered in our vSphere CSI driver documentation, which I will reproduce here:
- Use an existing storage device: You have provisioned a persistent storage, First Class Disk (FCD), directly in your vCenter, and want to use this FCD in your cluster.
- Make retained data available to the cluster: You have provisioned a volume with a reclaimPolicy:retain parameter in the storage class by using dynamic provisioning. You have removed the PVC, but the PV, the physical storage in vCenter, and the data still exists. You want to access the retained data from an application in your cluster.
- Share persistent storage across namespaces in the same cluster: You have provisioned a PV in a namespace of your cluster. You want to use the same storage instance for an application pod that is deployed to a different namespace in your cluster.
- Share persistent storage across clusters in the same zone: You have provisioned a PV for your cluster. To share the same persistent storage instance with other clusters in the same zone, you must manually create the PV and matching PVC in the other cluster.
Step 1: Create Volume in Volume Service
The very first step in the process is to use VCF Automation to create a volume. Here is an example of such a volume. The name of the volume is important as that is used to get the volume into VKS. Also note that this volume has a volume mode of Read-Write-Once, meaning that it will be created as a “block” volume.
Step 2: (Optional) Check the volume on the Supervisor
As highlighted, this is an optional step. But if you do have access to the Supervisor, you can check that the volume is present on the Supervisor after it has been created via the Volume Service. Note that the PVC (Persistent Volume Claim) will be associated with the Namespace where the volume from the Volume Service was created. Use the ‘-A’ option to the kubectl command to display all PVCs.
# kubectl get pvc -A NAMESPACE NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS VOLUMEATTRIBUTESCLASS AGE cjh-ns-01-jn88b volume-claim-cjh Bound pvc-0d16185f-626b-4788-a55e-4d3e48245226 7Gi RWO vsan-no-spaces <unset> 5s
Step 3: Build YAML manifests for PVC, PV and Pod
Next, create the following manifests in YAML, and apply them to the VKS cluster. The PV entry spec.csi.volumeHandle must include the PVC name retrieved from either the Supervisor or the VCFA Volume Service, i.e. “volume-claim-cjh”. Set the spec.accessMode to ReadWriteOnce and the spec.capacity.storage to match the size of the volume.
PV
apiVersion: v1 kind: PersistentVolume metadata: name: static-vks-block-pv annotations: pv.kubernetes.io/provisioned-by: csi.vsphere.vmware.com spec: storageClassName: vsan-no-spaces capacity: storage: 7Gi accessModes: - ReadWriteOnce persistentVolumeReclaimPolicy: Delete claimRef: namespace: default name: static-vks-block-pvc csi: driver: "csi.vsphere.vmware.com" volumeAttributes: type: "vSphere CNS Block Volume" volumeHandle: "volume-claim-cjh"
PVC
For the PVC manifest, again ensure that the spec.accessMode to set to ReadWriteOnce and that the spec.capacity.storage is also set to match the size of the volume. The spec.volumeName should be set to the name of the PV.
apiVersion: v1 kind: PersistentVolumeClaim apiVersion: v1 metadata: name: static-vks-block-pvc spec: accessModes: - ReadWriteOnce resources: requests: storage: 7Gi storageClassName: vsan-no-spaces volumeName: static-vks-block-pv
Pod
Finally, create a Pod so that we can confirm that the volume can be accessed successfully. Include a volume requirement which matches the PVC in spec. volumes.persistentVolumeClaim.claimName. This Pod has a very simply busybox container, but a number of spec.containers.securityContext settings are required for the Pod to successfully start on a VKS cluster.
apiVersion: v1 kind: Pod metadata: name: block-pod spec: containers: - name: busybox image: "registry.k8s.io/busybox:latest" volumeMounts: - name: block-vol mountPath: "/demo" command: [ "sleep", "1000000" ] securityContext: allowPrivilegeEscalation: false capabilities: drop: ["ALL"] runAsNonRoot: true runAsUser: 1000 seccompProfile: type: "RuntimeDefault" volumes: - name: block-vol persistentVolumeClaim: claimName: static-vks-block-pvc
Step 4: Verify Pod has access to volume
After applying these YAML manifests to the VKS cluster API server using kubectl, and assuming they create successfully, we can now verify that the volume is available in the Pod.
$ kubectl get pvc,pv,pod NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS VOLUMEATTRIBUTESCLASS AGE persistentvolumeclaim/static-vks-block-pvc Bound static-vks-block-pv 7Gi RWO vsan-no-spaces <unset> 20m NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS VOLUMEATTRIBUTESCLASS REASON AGE persistentvolume/static-vks-block-pv 7Gi RWO Delete Bound default/static-vks-block-pvc vsan-no-spaces <unset> 20m NAME READY STATUS RESTARTS AGE pod/block-pod 1/1 Running 0 11m $ kubectl exec -it block-pod -- "/bin/sh" / $ df -h Filesystem Size Used Available Use% Mounted on overlay 19.5G 6.6G 11.9G 36% / tmpfs 64.0M 0 64.0M 0% /dev /dev/sdb 6.8G 24.0K 6.4G 0% /demo /dev/sda3 19.5G 6.6G 11.9G 36% /etc/hosts /dev/sda3 19.5G 6.6G 11.9G 36% /dev/termination-log /dev/sda3 19.5G 6.6G 11.9G 36% /etc/hostname /dev/sda3 19.5G 6.6G 11.9G 36% /tmp/resolv.conf shm 64.0M 0 64.0M 0% /dev/shm tmpfs 13.0G 12.0K 13.0G 0% /tmp/secrets/kubernetes.io/serviceaccount tmpfs 7.8G 0 7.8G 0% /proc/acpi tmpfs 64.0M 0 64.0M 0% /proc/kcore tmpfs 64.0M 0 64.0M 0% /proc/keys tmpfs 64.0M 0 64.0M 0% /proc/latency_stats tmpfs 64.0M 0 64.0M 0% /proc/timer_list tmpfs 7.8G 0 7.8G 0% /proc/scsi tmpfs 7.8G 0 7.8G 0% /sys/firmware / $
This looks good. We have successfully provisioned a volume via the Volume Service in VCF Automation, and have been able to consume that volume by assigning it to a Pod in a VKS cluster. Great!
ReadWriteMany (RWX) Volumes
Now, you can also use the Volume Service in VCF Automation to provision a Read-Write-Many volume which can be shared by multiple Pods in a VKS cluster. Read-Write-Many volumes have many use-cases. I was able to find these by doing a quick google search.
- Shared File Systems and Collaboration Tools: Applications like content management systems, shared document repositories, or collaborative editing tools where multiple users or processes need concurrent read and write access to a common dataset.
- Distributed Caching: Shared caches used by multiple application instances to improve performance by storing frequently accessed data in a central location accessible to all.
- Centralized Logging and Monitoring Data: Storing logs or monitoring metrics from various application components in a central location where multiple tools or services might need to write data and multiple analysis tools need to read from it.
- Machine Learning Workloads: Where large datasets need to be accessed and potentially updated by multiple training or inference pods concurrently.
- Legacy Applications: When migrating older applications that rely heavily on shared file systems for data storage and cannot easily be refactored to use distributed databases or object storage.
- Clustered Applications: Certain clustered applications that require a shared file system for configuration, state, or data synchronization across multiple instances.
The one thing to be aware of is that RWX volumes created by the Volume Service in VCFA can only be consumed by Pods in VKS currently. It cannot be consumed by VMs created via the VM Service at this time. Note also that since RWX volumes are back by vSAN File Shares in VCF 9.0, you will need to have vSAN File Service enabled and configured in the Supervisor. You will also have to tell the Supervisor that you wish to consume vSAN File Services. Once those requirements are met, follow the steps outlined above, changing the accessMode from ReadWriteOnce to ReadWriteMany in the specifications. It should then be possibles for multiple Pods in a VKS cluster to access the same volume created by the Volume Service.
Thank you for reading this far. As always, reach out if there are any questions regarding this blog post.