CNS – not just for vSAN

by Posted on

After a very eventful VMworld, we received lots of questions about CNS, the Cloud Native Storage feature that was released with vSphere 6.7U3. Whilst most of the demonstrations and blog articles around CNS focused on vSAN, what may have been missed is that this feature also works with both VMFS and NFS datastores. For that reason, I decided to create some examples of how CNS can also bubble up information in vSphere about Kubernetes Persistent Volumes (PVs) created on both VMFS and NFS datastores. Let’s begin by creating some simple policies to tag my VMFS datastore and my NFS datastore. The policies will then form part of my StorageClass in Kubernetes. After deploying an app with Persistent Volumes (PVs) in Kubernetes, we will see what useful information about the PVs are displayed in the vSphere Client UI, giving a VI-Admin insight into how Kubernetes is consuming vSphere storage.

Tag the datastores

Before we begin, we first tag the VMFS and NFS datastores, since we will use tag based placement rule policies for this example. For simplicity, I created a tag category called Storage, tagged an NFS datastore as bronze-nfs, and tagged a VMFS-6 datastore as silver-vmfs. Here are a couple of screenshots showing the tagging:

Now that we have tagged the datastores, we can go ahead and use Storage Policy Based Management (SPBM) to create two tag based policies.

Build the policies

Creating the policies is very straight forward. Under VM Storage Policies in the vSphere Client UI, click on the option to ‘Create VM Storage Policy’. From there, chose the option to do tag based placement rules, as shown below:

I proceeded to create two policies, one policy which would resolve to my NFS datastore and one policy which would resolve to my VMFS-6 datastore. By choosing the correct tag category (Storage) and appropriate tag (silver-vmfs or bronze-nfs), I now have a policy (called bronze) which would select the NFS datastore for placement, and another policy (called silver) which would select the VMFS datastore for placement. The policy details are shown below.

These policy can now be references through the StorageClass construct in Kubernetes.

Create a StorageClass to match policy

Now that the policies exist, the vSphere CSI (Container Storage Interface) driver allows StorageClass construct in Kubernetes to reference SPBM policies simply by adding a parameter.storagepolicyname in the StorageClass manifest (yaml). Here is a sample StorageClass that I used for the silver/VMFS policy:

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: cass-vmfs-sc
annotations:
storageclass.kubernetes.io/is-default-class: "false"
provisioner: csi.vsphere.vmware.com
parameters:
storagepolicyname: "silver"

The bronze/NFS policy is almost identical:

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: cass-nfs-sc
annotations:
storageclass.kubernetes.io/is-default-class: "false"
provisioner: csi.vsphere.vmware.com
parameters:
storagepolicyname: "bronze"

Any persistent volumes created with the above StorageClasses will be provisioned on either VMFS or NFS, depending on whether the StorageClass referenced is silver or bronze respectively.

So what does this look like in CNS?

Cloud Native Storage for NFS /VMFS

Again, once you are using vSphere 6.7U3 and the new CSI driver, persistent volumes provisioned on VMFS or NFS using the StorageClasses created above will also show up in the CNS UI. For illustration purposes, I deployed a 3-node Cassandra application on both NFS and VMFS in the same K8s cluster. This is what it looked like in CNS:

Here we can see label information (bubbled up from Kubernetes), the datastore, the policy (silver or bronze), whether the policy is compliant or not, and various other information. We can also click on the icon in the second column for more details about the PV:

And here we can see the persistent volume and persistent volume claim details, including the namespace and the Pod to which the PV is attached. The Basics view provides more policy related info, but also which Kubernetes worker node the PV is attached to (note that on vSphere storage, the PV can be thought of simply as a VMDK):

In conclusion, I hope this demonstrates how CNS not only provides insights to Kubernetes persistent volumes deployed on vSAN but also on other vSphere storage types, such as VMFS and NFS. We hope to have CNS supporting vVols in the very near future.

Published by Cormac