vSAN File Service & Kubernetes PVs with an implicit quota

Earlier this week, I participated in a customer call around vSAN File Service and Kubernetes Persistent Volumes. I have highlighted the dynamic Read-Write-Many Persistent Volume feature of our vSphere CSI driver in conjunction with vSAN File Service before. Read-Write-Many (RWX) volumes are volumes that can be accessed/shared by multiple containers. During the discussion, one question came up in relation to quota, and if it can be applied to Persistent Volumes which are backed by file shares from vSAN File Service, which is the purpose of this post. Now, for those of you who are familiar with vSAN File Service, you will be aware that when creating a standard file share via the vSphere UI, there is the opportunity to set a Share warning threshold and a Share hard quota, e.g.

If you do not set a hard quota, then theoretically the full capacity of the underlying datastore is available via the share. Indeed, if you examine the client where the file share is mounted, you will observe the full capacity of the underlying datastore is available, in this case 2TB. Below is an example where a file share is created with no quota and mounted to a client machine. Some random data is then copied to it. It will continue to store data without complaint, until the underlying datastore is filled (if allowed to do so – I control-c’ed the operation):

$ df -h
Filesystem                                      Size  Used Avail Use% Mounted on
udev                                            3.9G    0  3.9G  0% /dev
tmpfs                                           798M  1.7M  797M  1% /run
/dev/sda1                                        16G   12G  3.6G  77% /
tmpfs                                           3.9G  8.0K  3.9G  1% /dev/shm
tmpfs                                           5.0M    0  5.0M  0% /run/lock
tmpfs                                           3.9G    0  3.9G  0% /sys/fs/cgroup
/dev/sdb1                                       98G   4.2G  89G  5% /usr/share/go
/dev/sde1                                      196G    13G  173G  7% /var/lib/docker
/dev/sdc1                                       49G    53M  47G  1% /home/linuxbrew
/dev/sdd1                                       98G   478M  93G  1% /opt
/dev/sdf1                                      196G   6.9G  179G  4% /home/cormac
tmpfs                                          798M   136K  798M  1% /run/user/1000
vsan-fs1.rainpole.com:/new-nfs-share-no-quota  2.0T     0  2.0T  0% /vsanfs-noquota

$ pwd
/vsanfs-noquota

$ sudo cp -r / .
^C

$ df -h
Filesystem                                     Size   Used Avail Use% Mounted on
udev                                           3.9G     0   3.9G   0% /dev
tmpfs                                          798M   1.7M  797M   1% /run
/dev/sda1                                       16G    12G  3.6G  77% /
tmpfs                                          3.9G   8.0K  3.9G   1% /dev/shm
tmpfs                                          5.0M     0   5.0M   0% /run/lock
tmpfs                                          3.9G     0   3.9G   0% /sys/fs/cgroup
/dev/sdb1                                       98G   4.2G   89G   5% /usr/share/go
/dev/sde1                                      196G   13G   173G   7% /var/lib/docker
/dev/sdc1                                       49G   53M    47G   1% /home/linuxbrew
/dev/sdd1                                       98G  478M    93G   1% /opt
/dev/sdf1                                      196G  6.9G   179G   4% /home/cormac
tmpfs                                          798M  136K   798M   1% /run/user/1000
vsan-fs1.rainpole.com:/new-nfs-share-no-quota  2.0T  2.4G   2.0T   1% /vsanfs-noquota

Let’s try the same exercise with a file share that has quota. After creating the quota with the warning threshold and hard quota set, the Usage/Quota field is now populated in the UI, unlike the share with no quota set:

I will now mount this new share with a quota to the same client where the previous share with no quota was mounted, and attempt to copy some data to that share. Note however that the available capacity of the share still reflects all of the capacity on the underlying datastore where the share resides. Now as we shall see, with the quota in place, it only allows the user to consume only up to the quota that has been allocated to the share, not the whole datastore on the back-end.

$ df
Filesystem                                      1K-blocks    Used  Available   Use% Mounted on
udev                                              4057224        0    4057224    0% /dev
tmpfs                                              816876     1708     815168    1% /run
/dev/sda1                                        16446332  11898700   3692492   77% /
tmpfs                                              4084364        8   4084356    1% /dev/shm
tmpfs                                                5120         0      5120    0% /run/lock
tmpfs                                              4084364        0    4084364   0% /sys/fs/cgroup
/dev/sdb1                                        102686648  4315256   93112180   5% /usr/share/go
/dev/sde1                                        205374440 13503732  181368616   7% /var/lib/docker
/dev/sdc1                                        51342816     53276   48651768   1% /home/linuxbrew
/dev/sdd1                                        102686648   488552   96938884   1% /opt
/dev/sdf1                                        205374440  7191760  187680588   4% /home/cormac
tmpfs                                              816872       136     816736   1% /run/user/1000
vsan-fs1.rainpole.com:/new-nfs-share-no-quota   2113368064  2441216 2110824448   1% /vsanfs-noquota
vsan-fs1.rainpole.com:/new-nfs-share-with-quota 2111035392   108544  2110824448  1% /vsanfs-with-quota

$ cd /vsanfs-with-quota

$ sudo cp -r /vsanfs-noquota/* .
^C

$ df -h
Filesystem                                      Size   Used Avail Use% Mounted on
udev                                             3.9G     0  3.9G  0% /dev
tmpfs                                            798M  1.7M  797M  1% /run
/dev/sda1                                        16G    12G  3.6G  77% /
tmpfs                                            3.9G  8.0K  3.9G  1% /dev/shm
tmpfs                                            5.0M     0  5.0M  0% /run/lock
tmpfs                                            3.9G     0  3.9G  0% /sys/fs/cgroup
/dev/sdb1                                         98G  4.2G   89G  5% /usr/share/go
/dev/sde1                                        196G   13G  173G  7% /var/lib/docker
/dev/sdc1                                         49G   53M   47G  1% /home/linuxbrew
/dev/sdd1                                         98G  478M   93G  1% /opt
/dev/sdf1                                        196G  6.9G  179G  4% /home/cormac
tmpfs                                            798M  136K  798M  1% /run/user/1000
vsan-fs1.rainpole.com:/new-nfs-share-no-quota    2.0T  2.4G  2.0T  1% /vsanfs-noquota
vsan-fs1.rainpole.com:/new-nfs-share-with-quota  2.0T  1.7G  2.0T  1% /vsanfs-with-quota

And as we fill the share, we can monitor the amount of space consumed via the vSphere client:

If the capacity used reaches the configured warning threshold, we see this highlighted in the UI:

If we continue to add data and the capacity usage reaches the hard quota, it is not possible to add more data to the share with the quota:

$ sudo cp -r /vsanfs-noquota/* .
cp: failed to close './usr/bin/docker': Disk quota exceeded
cp: failed to close './usr/bin/activity-log-manager': Disk quota exceeded
cp: failed to close './usr/bin/gnome-software': Disk quota exceeded
cp: failed to close './usr/bin/x86_64-linux-gnu-cpp-6': Disk quota exceeded
cp: failed to close './usr/bin/ssh-keyscan': Disk quota exceeded
cp: failed to close './usr/bin/groff': Disk quota exceeded
^C
$

So that covers standard volumes. What about Kubernetes Persistent Volumes that have been dynamically provisioned by the vSphere CSI driver using vSAN File Service? Well, basically the storage request in the Persistent Volume Claim (PVC) manifest behaves as a hard quota. For example, here is a sample PVC manifest that is leveraging a StorageClass which has been configured to use vSAN File Services. This claim is requesting a RWX volume of size 5GB.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: file-pvc
spec:
  accessModes:
  - ReadWriteMany
  resources:
    requests:
      storage: 5Gi
  storageClassName: vsan-file

As part of this test, I also deployed some Pods to consume this RWX volume. Below, I am exec’ing into one of those Pods to see if I can write more than the 5GB requested for this persistent volume.

% kubectl get pods
NAME                            READY  STATUS    RESTARTS  AGE
app-1                            1/1    Running  0          6h57m
app-2                            1/1    Running  0          6h57m

% kubectl exec -it app-1 -- sh
/ # df -h
Filesystem                Size      Used Available Use% Mounted on
overlay                  58.0G      7.7G    50.2G  13% /
tmpfs                    64.0M        0    64.0M  0% /dev
tmpfs                    1.9G        0      1.9G  0% /sys/fs/cgroup
/dev/sda1                58.0G      7.7G    50.2G  13% /dev/termination-log
vsan-fs1.rainpole.com:/52d35578-2f50-44f7-c3c7-37271064864d
                          2.0T        0      2.0T  0% /mnt/volume1
/dev/sda1                58.0G      7.7G    50.2G  13% /etc/resolv.conf
/dev/sda1                58.0G      7.7G    50.2G  13% /etc/hostname
/dev/sda1                58.0G      7.7G    50.2G  13% /etc/hosts
shm                      64.0M        0    64.0M  0% /dev/shm
tmpfs                    1.9G    12.0K      1.9G  0% /var/run/secrets/kubernetes.io/serviceaccount
tmpfs                    1.9G        0      1.9G  0% /proc/acpi
tmpfs                    64.0M        0    64.0M  0% /proc/kcore
tmpfs                    64.0M        0    64.0M  0% /proc/keys
tmpfs                    64.0M        0    64.0M  0% /proc/timer_list
tmpfs                    64.0M        0    64.0M  0% /proc/sched_debug
tmpfs                    1.9G        0      1.9G  0% /proc/scsi
tmpfs                    1.9G        0      1.9G  0% /sys/firmware

If we examine this persistent volume via the vSphere client, we can see that it appears to have an implicit quota associated with it (the Usage/Quota field)

And now if we copy in data to it, we should be prevented from copying any more data that the 5GB which is what we specified in the Persistent Volume Claim manifest file.

/ # cd /mnt/volume1/
/mnt/volume1 # cp -r /* .
cp: error writing to './bin/setlogcons': Disk quota exceeded
cp: error writing to './mnt/volume1/proc/dynamic_debug/control': Disk quota exceeded
cp: error writing to './mnt/volume1/proc/kpagecgroup': Disk quota exceeded
cp: error writing to './mnt/volume1/proc/7638/pagemap': Disk quota exceeded
cp: error writing to './mnt/volume1/proc/7638/task/7638/pagemap': Disk quota exceeded
cp: error writing to './mnt/volume1/proc/kpagecount': Disk quota exceeded
cp: error writing to './mnt/volume1/proc/vmallocinfo': Disk quota exceeded
cp: error writing to './mnt/volume1/proc/kpageflags': Disk quota exceeded

And the usage is also visible in the vSphere client:

So this has hopefully demonstrated how dynamic RWX Persistent Volumes, built on vSAN File Service File Shares, have implicit quotas set which match the size specified in the Persistent Volume Claim manifest. Note that if more capacity is required for a persistent volume, there is an online volume expand option available in the vSphere CSI driver. More details about online volume expand can be found here.