Read-Only Persistent Volumes on vSAN File Services
I’m writing this post because of a misconception I had regarding how read-only volumes were configured in Kubernetes. I thought this was controlled by the accessModes parameter in the PersistentVolumeClaim manifest file. This is not the case. It is controlled from the Pod, which to me seems a bit strange. Why would this not be controlled from the PVC manifest? One of our engineers pointed me to a few Kubernetes discussions on the behaviour of accessModes and readOnly here and here. It would seem that I am not the only one confused by this behaviour. In this post, I deploy a dynamic, file-based Persistent Volume from vSAN File Services, and then deploy 2 Pods. One pod has read-write access to the volume while the other has read-only access. Then we will see what they can do on the file share which is presented to both Pods.
Here are the manifests. Note that the PVC manifest has the accessModes set to ReadOnlyMany, but this does not prevent the Pods from writing to the PV. The spec.volumes.persistentVolumeClaim.readOnly:true parameter in the Pod manifests is what controls the access.
apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: vsan-file-sc provisioner: csi.vsphere.vmware.com parameters: storagepolicyname: "RAID1" csi.storage.k8s.io/fstype: nfs4 apiVersion: v1 kind: PersistentVolumeClaim metadata: name: file-pvc-rox spec: storageClassName: vsan-file-sc accessModes: - ReadOnlyMany resources: requests: storage: 2Gi apiVersion: v1 kind: Pod metadata: name: file-pod-a-rox spec: containers: - name: file-pod-a-rox image: "k8s.gcr.io/busybox" volumeMounts: - name: file-vol-rox mountPath: "/mnt/volume1" command: [ "sleep", "1000000" ] volumes: - name: file-vol-rox persistentVolumeClaim: claimName: file-pvc-rox readOnly: true apiVersion: v1 kind: Pod metadata: name: file-pod-b-rwx spec: containers: - name: file-pod-b-rwx image: "k8s.gcr.io/busybox" volumeMounts: - name: file-vol-b-rwx mountPath: "/mnt/volume1" command: [ "sleep", "1000000" ] volumes: - name: file-vol-b-rwx persistentVolumeClaim: claimName: file-pvc-rox readOnly: false
Let’s now go ahead and create the Kubernetes objects (StorageClass, PV, PVC and Pods). First l deploy a Pod that has volume readOnly set to false, and here we see that you can still write to the shared volume (implying that the accessModes setting in the PVC manifest is not having any impact).
$ kubectl apply -f fs-pod-rw-b.yaml pod/file-pod-b-rwx created $ kubectl get pod NAME READY STATUS RESTARTS AGE file-pod-b-rwx 1/1 Running 0 5s $ kubectl exec -it file-pod-b-rwx -- /bin/sh / # cd /mnt/volume1/ /mnt/volume1 # mkdir demo-test /mnt/volume1 # cd demo-test/ /mnt/volume1/demo-test # while true > do > date >> datestamp > sleep 5 > done
Absolutely no issues writing to the volume from this Pod. I am able to create a directory and file on the shared volume, and also write to the file. Let’s now deploy the other Pod which has spec.volumes.persistentVolumeClaim.readOnly set to true.
$ kubectl apply -f fs-pod-ro-a.yaml
pod/file-pod-a-rox created
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
file-pod-a-rox 0/1 ContainerCreating 0 7s
file-pod-b-rwx 1/1 Running 0 101s
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
file-pod-a-rox 1/1 Running 0 13s
file-pod-b-rwx 1/1 Running 0 107s
$ kubectl exec -it file-pod-a-rox -- /bin/sh
/ # cd /mnt/volume1/
/mnt/volume1 # ls
demo-test
/mnt/volume1 # mkdir new-test
mkdir: can't create directory 'new-test': Read-only file system
/mnt/volume1 # cd demo-test/
/mnt/volume1/demo-test # tail -f datestamp
Wed Jun 3 15:19:08 UTC 2020
Wed Jun 3 15:19:13 UTC 2020
Wed Jun 3 15:19:18 UTC 2020
Wed Jun 3 15:19:23 UTC 2020
Wed Jun 3 15:19:28 UTC 2020
Wed Jun 3 15:19:33 UTC 2020
Wed Jun 3 15:19:38 UTC 2020
Wed Jun 3 15:19:43 UTC 2020
Wed Jun 3 15:19:48 UTC 2020
Wed Jun 3 15:19:53 UTC 2020
In this Pod, we cannot write to the volume. An attempt to create a new directory failed, stating that the file system is read-only. However we can successfully read from the share, as we can read the contents of the file that is being written to by the other Pod sharing the volume. Therefore, if you want read-only access for a volume, it must be implemented in the Pod manifest via spec.volumes.persistentVolumeClaim.readOnly. The accessModes parameter in the PVC does not control it.
Cormac,
as I see it, a PVC is a binding from a container to a Volume. It’s a claim.
I would expect that the binding would fail if the volume was created readOnly and the claim was set to RW.
But if you ask for just R/O and you are given a R/W capable… you still can write to it. 🙂
I agree that this is not at all intuitive. But this seems to be how Kubernetes are dealing with NFS in general. See the links to discussions on the topic in the early part of the post. I’d like to it change at some point, and make it more logical.