vSAN File Service & Kubernetes PVs with an implicit quota
Earlier this week, I participated in a customer call around vSAN File Service and Kubernetes Persistent Volumes. I have highlighted the dynamic Read-Write-Many Persistent Volume feature of our vSphere CSI driver in conjunction with vSAN File Service before. Read-Write-Many (RWX) volumes are volumes that can be accessed/shared by multiple containers. During the discussion, one question came up in relation to quota, and if it can be applied to Persistent Volumes which are backed by file shares from vSAN File Service, which is the purpose of this post. Now, for those of you who are familiar with vSAN File Service, you will be aware that when creating a standard file share via the vSphere UI, there is the opportunity to set a Share warning threshold and a Share hard quota, e.g.
If you do not set a hard quota, then theoretically the full capacity of the underlying datastore is available via the share. Indeed, if you examine the client where the file share is mounted, you will observe the full capacity of the underlying datastore is available, in this case 2TB. Below is an example where a file share is created with no quota and mounted to a client machine. Some random data is then copied to it. It will continue to store data without complaint, until the underlying datastore is filled (if allowed to do so – I control-c’ed the operation):
$ df -h Filesystem Size Used Avail Use% Mounted on udev 3.9G 0 3.9G 0% /dev tmpfs 798M 1.7M 797M 1% /run /dev/sda1 16G 12G 3.6G 77% / tmpfs 3.9G 8.0K 3.9G 1% /dev/shm tmpfs 5.0M 0 5.0M 0% /run/lock tmpfs 3.9G 0 3.9G 0% /sys/fs/cgroup /dev/sdb1 98G 4.2G 89G 5% /usr/share/go /dev/sde1 196G 13G 173G 7% /var/lib/docker /dev/sdc1 49G 53M 47G 1% /home/linuxbrew /dev/sdd1 98G 478M 93G 1% /opt /dev/sdf1 196G 6.9G 179G 4% /home/cormac tmpfs 798M 136K 798M 1% /run/user/1000 vsan-fs1.rainpole.com:/new-nfs-share-no-quota 2.0T 0 2.0T 0% /vsanfs-noquota $ pwd /vsanfs-noquota $ sudo cp -r / . ^C $ df -h Filesystem Size Used Avail Use% Mounted on udev 3.9G 0 3.9G 0% /dev tmpfs 798M 1.7M 797M 1% /run /dev/sda1 16G 12G 3.6G 77% / tmpfs 3.9G 8.0K 3.9G 1% /dev/shm tmpfs 5.0M 0 5.0M 0% /run/lock tmpfs 3.9G 0 3.9G 0% /sys/fs/cgroup /dev/sdb1 98G 4.2G 89G 5% /usr/share/go /dev/sde1 196G 13G 173G 7% /var/lib/docker /dev/sdc1 49G 53M 47G 1% /home/linuxbrew /dev/sdd1 98G 478M 93G 1% /opt /dev/sdf1 196G 6.9G 179G 4% /home/cormac tmpfs 798M 136K 798M 1% /run/user/1000 vsan-fs1.rainpole.com:/new-nfs-share-no-quota 2.0T 2.4G 2.0T 1% /vsanfs-noquota
Let’s try the same exercise with a file share that has quota. After creating the quota with the warning threshold and hard quota set, the Usage/Quota field is now populated in the UI, unlike the share with no quota set:
I will now mount this new share with a quota to the same client where the previous share with no quota was mounted, and attempt to copy some data to that share. Note however that the available capacity of the share still reflects all of the capacity on the underlying datastore where the share resides. Now as we shall see, with the quota in place, it only allows the user to consume only up to the quota that has been allocated to the share, not the whole datastore on the back-end.
$ df Filesystem 1K-blocks Used Available Use% Mounted on udev 4057224 0 4057224 0% /dev tmpfs 816876 1708 815168 1% /run /dev/sda1 16446332 11898700 3692492 77% / tmpfs 4084364 8 4084356 1% /dev/shm tmpfs 5120 0 5120 0% /run/lock tmpfs 4084364 0 4084364 0% /sys/fs/cgroup /dev/sdb1 102686648 4315256 93112180 5% /usr/share/go /dev/sde1 205374440 13503732 181368616 7% /var/lib/docker /dev/sdc1 51342816 53276 48651768 1% /home/linuxbrew /dev/sdd1 102686648 488552 96938884 1% /opt /dev/sdf1 205374440 7191760 187680588 4% /home/cormac tmpfs 816872 136 816736 1% /run/user/1000 vsan-fs1.rainpole.com:/new-nfs-share-no-quota 2113368064 2441216 2110824448 1% /vsanfs-noquota vsan-fs1.rainpole.com:/new-nfs-share-with-quota 2111035392 108544 2110824448 1% /vsanfs-with-quota $ cd /vsanfs-with-quota $ sudo cp -r /vsanfs-noquota/* . ^C $ df -h Filesystem Size Used Avail Use% Mounted on udev 3.9G 0 3.9G 0% /dev tmpfs 798M 1.7M 797M 1% /run /dev/sda1 16G 12G 3.6G 77% / tmpfs 3.9G 8.0K 3.9G 1% /dev/shm tmpfs 5.0M 0 5.0M 0% /run/lock tmpfs 3.9G 0 3.9G 0% /sys/fs/cgroup /dev/sdb1 98G 4.2G 89G 5% /usr/share/go /dev/sde1 196G 13G 173G 7% /var/lib/docker /dev/sdc1 49G 53M 47G 1% /home/linuxbrew /dev/sdd1 98G 478M 93G 1% /opt /dev/sdf1 196G 6.9G 179G 4% /home/cormac tmpfs 798M 136K 798M 1% /run/user/1000 vsan-fs1.rainpole.com:/new-nfs-share-no-quota 2.0T 2.4G 2.0T 1% /vsanfs-noquota vsan-fs1.rainpole.com:/new-nfs-share-with-quota 2.0T 1.7G 2.0T 1% /vsanfs-with-quota
And as we fill the share, we can monitor the amount of space consumed via the vSphere client:
If the capacity used reaches the configured warning threshold, we see this highlighted in the UI:
If we continue to add data and the capacity usage reaches the hard quota, it is not possible to add more data to the share with the quota:
$ sudo cp -r /vsanfs-noquota/* . cp: failed to close './usr/bin/docker': Disk quota exceeded cp: failed to close './usr/bin/activity-log-manager': Disk quota exceeded cp: failed to close './usr/bin/gnome-software': Disk quota exceeded cp: failed to close './usr/bin/x86_64-linux-gnu-cpp-6': Disk quota exceeded cp: failed to close './usr/bin/ssh-keyscan': Disk quota exceeded cp: failed to close './usr/bin/groff': Disk quota exceeded ^C $
So that covers standard volumes. What about Kubernetes Persistent Volumes that have been dynamically provisioned by the vSphere CSI driver using vSAN File Service? Well, basically the storage request in the Persistent Volume Claim (PVC) manifest behaves as a hard quota. For example, here is a sample PVC manifest that is leveraging a StorageClass which has been configured to use vSAN File Services. This claim is requesting a RWX volume of size 5GB.
apiVersion: v1 kind: PersistentVolumeClaim metadata: name: file-pvc spec: accessModes: - ReadWriteMany resources: requests: storage: 5Gi storageClassName: vsan-file
As part of this test, I also deployed some Pods to consume this RWX volume. Below, I am exec’ing into one of those Pods to see if I can write more than the 5GB requested for this persistent volume.
% kubectl get pods NAME READY STATUS RESTARTS AGE app-1 1/1 Running 0 6h57m app-2 1/1 Running 0 6h57m % kubectl exec -it app-1 -- sh / # df -h Filesystem Size Used Available Use% Mounted on overlay 58.0G 7.7G 50.2G 13% / tmpfs 64.0M 0 64.0M 0% /dev tmpfs 1.9G 0 1.9G 0% /sys/fs/cgroup /dev/sda1 58.0G 7.7G 50.2G 13% /dev/termination-log vsan-fs1.rainpole.com:/52d35578-2f50-44f7-c3c7-37271064864d 2.0T 0 2.0T 0% /mnt/volume1 /dev/sda1 58.0G 7.7G 50.2G 13% /etc/resolv.conf /dev/sda1 58.0G 7.7G 50.2G 13% /etc/hostname /dev/sda1 58.0G 7.7G 50.2G 13% /etc/hosts shm 64.0M 0 64.0M 0% /dev/shm tmpfs 1.9G 12.0K 1.9G 0% /var/run/secrets/kubernetes.io/serviceaccount tmpfs 1.9G 0 1.9G 0% /proc/acpi tmpfs 64.0M 0 64.0M 0% /proc/kcore tmpfs 64.0M 0 64.0M 0% /proc/keys tmpfs 64.0M 0 64.0M 0% /proc/timer_list tmpfs 64.0M 0 64.0M 0% /proc/sched_debug tmpfs 1.9G 0 1.9G 0% /proc/scsi tmpfs 1.9G 0 1.9G 0% /sys/firmware
If we examine this persistent volume via the vSphere client, we can see that it appears to have an implicit quota associated with it (the Usage/Quota field)
And now if we copy in data to it, we should be prevented from copying any more data that the 5GB which is what we specified in the Persistent Volume Claim manifest file.
/ # cd /mnt/volume1/ /mnt/volume1 # cp -r /* . cp: error writing to './bin/setlogcons': Disk quota exceeded cp: error writing to './mnt/volume1/proc/dynamic_debug/control': Disk quota exceeded cp: error writing to './mnt/volume1/proc/kpagecgroup': Disk quota exceeded cp: error writing to './mnt/volume1/proc/7638/pagemap': Disk quota exceeded cp: error writing to './mnt/volume1/proc/7638/task/7638/pagemap': Disk quota exceeded cp: error writing to './mnt/volume1/proc/kpagecount': Disk quota exceeded cp: error writing to './mnt/volume1/proc/vmallocinfo': Disk quota exceeded cp: error writing to './mnt/volume1/proc/kpageflags': Disk quota exceeded
And the usage is also visible in the vSphere client:
So this has hopefully demonstrated how dynamic RWX Persistent Volumes, built on vSAN File Service File Shares, have implicit quotas set which match the size specified in the Persistent Volume Claim manifest. Note that if more capacity is required for a persistent volume, there is an online volume expand option available in the vSphere CSI driver. More details about online volume expand can be found here.
