Read-Only Persistent Volumes on vSAN File Services

I’m writing this post because of a misconception I had regarding how read-only volumes were configured in Kubernetes. I thought this was controlled by the accessModes parameter in the PersistentVolumeClaim manifest file. This is not the case. It is controlled from the Pod, which to me seems a bit strange. Why would this not be controlled from the PVC manifest? One of our engineers pointed me to a few Kubernetes discussions on the behaviour of accessModes and readOnly here and here. It would seem that I am not the only one confused by this behaviour. In this post, I deploy a dynamic, file-based Persistent Volume from vSAN File Services, and then deploy 2 Pods. One pod has read-write access to the volume while the other has read-only access. Then we will see what they can do on the file share which is presented to both Pods.

Here are the manifests. Note that the PVC manifest has the accessModes set to ReadOnlyMany, but this does not prevent the Pods from writing to the PV. The spec.volumes.persistentVolumeClaim.readOnly:true parameter in the Pod manifests is what controls the access.

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: vsan-file-sc
provisioner: csi.vsphere.vmware.com
parameters:
  storagepolicyname: "RAID1"
  csi.storage.k8s.io/fstype: nfs4


apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: file-pvc-rox
spec:
  storageClassName: vsan-file-sc
  accessModes:
    - ReadOnlyMany
  resources:
    requests:
      storage: 2Gi


apiVersion: v1
kind: Pod
metadata:
  name: file-pod-a-rox
spec:
  containers:
  - name: file-pod-a-rox
    image: "k8s.gcr.io/busybox"
    volumeMounts:
    - name: file-vol-rox
      mountPath: "/mnt/volume1"
    command: [ "sleep", "1000000" ]
  volumes:
    - name: file-vol-rox
      persistentVolumeClaim:
        claimName: file-pvc-rox
        readOnly: true


apiVersion: v1
kind: Pod
metadata:
  name: file-pod-b-rwx
spec:
  containers:
  - name: file-pod-b-rwx
    image: "k8s.gcr.io/busybox"
    volumeMounts:
    - name: file-vol-b-rwx
      mountPath: "/mnt/volume1"
    command: [ "sleep", "1000000" ]
  volumes:
    - name: file-vol-b-rwx
      persistentVolumeClaim:
        claimName: file-pvc-rox
        readOnly: false

 Let’s now go ahead and create the Kubernetes objects (StorageClass, PV, PVC and Pods). First l deploy a Pod that has volume readOnly set to false, and here we see that you can still write to the shared volume (implying that the accessModes setting in the PVC manifest is not having any impact).

$ kubectl apply -f fs-pod-rw-b.yaml
pod/file-pod-b-rwx created


$ kubectl get pod
NAME             READY   STATUS    RESTARTS   AGE
file-pod-b-rwx   1/1     Running   0          5s


$ kubectl exec -it file-pod-b-rwx -- /bin/sh
/ # cd /mnt/volume1/
/mnt/volume1 # mkdir demo-test
/mnt/volume1 # cd demo-test/
/mnt/volume1/demo-test # while true
> do
> date >> datestamp
> sleep 5
> done

Absolutely no issues writing to the volume from this Pod. I am able to create a directory and file on the shared volume, and also write to the file. Let’s now deploy the other Pod which has spec.volumes.persistentVolumeClaim.readOnly set to true.

$ kubectl apply -f fs-pod-ro-a.yaml
pod/file-pod-a-rox created


$ kubectl get pods
NAME             READY   STATUS              RESTARTS   AGE
file-pod-a-rox   0/1     ContainerCreating   0          7s
file-pod-b-rwx   1/1     Running             0          101s


$ kubectl get pods
NAME             READY   STATUS    RESTARTS   AGE
file-pod-a-rox   1/1     Running   0          13s
file-pod-b-rwx   1/1     Running   0          107s


$ kubectl exec -it file-pod-a-rox -- /bin/sh
/ # cd /mnt/volume1/
/mnt/volume1 # ls
demo-test
/mnt/volume1 # mkdir new-test
mkdir: can't create directory 'new-test': Read-only file system
/mnt/volume1 # cd demo-test/
/mnt/volume1/demo-test # tail -f datestamp
Wed Jun  3 15:19:08 UTC 2020
Wed Jun  3 15:19:13 UTC 2020
Wed Jun  3 15:19:18 UTC 2020
Wed Jun  3 15:19:23 UTC 2020
Wed Jun  3 15:19:28 UTC 2020
Wed Jun  3 15:19:33 UTC 2020
Wed Jun  3 15:19:38 UTC 2020
Wed Jun  3 15:19:43 UTC 2020
Wed Jun  3 15:19:48 UTC 2020
Wed Jun  3 15:19:53 UTC 2020

In this Pod, we cannot write to the volume. An attempt to create a new directory failed, stating that the file system is read-only. However we can successfully read from the share, as we can read the contents of the file that is being written to by the other Pod sharing the volume. Therefore, if you want read-only access for a volume, it must be implemented in the Pod manifest via spec.volumes.persistentVolumeClaim.readOnly. The accessModes parameter in the PVC does not control it.

2 Replies to “Read-Only Persistent Volumes on vSAN File Services”

  1. Cormac,
    as I see it, a PVC is a binding from a container to a Volume. It’s a claim.
    I would expect that the binding would fail if the volume was created readOnly and the claim was set to RW.
    But if you ask for just R/O and you are given a R/W capable… you still can write to it. 🙂

    1. I agree that this is not at all intuitive. But this seems to be how Kubernetes are dealing with NFS in general. See the links to discussions on the topic in the early part of the post. I’d like to it change at some point, and make it more logical.

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.