[Update] Before going any further, it is worth highlighting the differences between what we term native Kubernetes and vSphere with Kubernetes. Native Kubernetes has many flavors, such as VMware Tanzu Kubernetes Grid, VMware Tanzu Kubernetes Grid Integrated (TKGI) formerly known as Enterprise PKS, RedHat OpenShift and Google Anthos. We refer to the vSphere CSI driver used with these native Kubernetes flavors as the upstream vSphere CSI driver.
In vSphere with Kubernetes, there are new optimized CSI drivers for both the Supervisor cluster and the Tanzu Kubernetes Grid “guest” cluster. The upstream vSphere CSI driver has been engineered to run on the supervisor, and engineered to run in a paravirtualized fashion on TKG “guest” cluster. Storage requests from applications deployed on TKG “guest” clusters are routed through the supervisor cluster. We refer to the vSphere CSI driver in the TKG “guest” clusters as pvCSI.
Since vSphere with Kubernetes is only available with VCF 4.0 at this time, it also implies that vSphere 7.0 is required. However, the vSphere with Kubernetes CSI drivers do not have the same set of features as the upstream CSI driver running in a native K8s cluster which is deployed onto vSphere 7.0. So while the vSphere CSI drivers in vSphere with Kubernetes and TKG “guest” clusters are based off of the upstream CSI, we may see a lag in the CSI features on this platform due to the additional optimizations needed to run the CSI driver.
This post is aimed at highlighting the different versions of the drivers, and their different respective capabilities. Tanzu Kubernetes Grid Integrated (TKGI) Edition, formerly known as Enterprise PKS, also supports the CSI driver in version 1.7 which was released last month. Note that TKGI/Enterprise PKS 1.7 also has the earlier VCP driver available as an option as well. This may have different features and functionality, but in this post I am focusing on the features available in Enterprise PKS 1.7 with the CSI driver only.
| Native K8s on vSphere 6.7U3
(CSI 1.0.2 & 2.0) |
Native K8s on vSphere 7.0
(CSI 2.0) |
vSphere with Kubernetes – Supervisor Cluster (CSI 2.0) |
vSphere with Kubernetes – TKG ‘Guest’ Cluster (CSI 2.0) |
Enterprise PKS 1.7 (TKGI) on vSphere 6.7U3 (CSI 1.0.2) |
|
| CNS UI Support |
Yes |
Yes |
Yes |
Yes |
Yes |
| Enhanced Object Health in UI |
Yes (vSAN only) | Yes (vSAN only) | Yes (vSAN only) | Yes (vSAN only) | Yes (vSAN only) |
| Dynamic Block PV support (Read-Write-Once Access Mode) | Yes | Yes | Yes | Yes | Yes |
| Dynamic File PV support (Read-Write-Many Access Mode) | No | Yes (vSAN only) | No | No | No |
| Encryption support via VMcrypt | No | Yes (Block PV) | No | No | No |
| Dynamic Virtual Volume PV support | No | Yes | No | No | No |
| Offline Volume Grow support (beta) |
No | Yes (Block PV) | No | No | No |
| Topology/Availability Zone support | Yes (Block PV) |
Yes (Block PV) |
No | No | No |
| Static PV Provisioning | Yes | Yes | No | Yes | Yes |
| K8s Multi-node Control Plane support |
Yes |
Yes |
Yes |
Yes |
Yes |
| WaitForConsumerFirst |
Yes |
Yes |
No |
No |
Yes |
Additional notes:
- vSphere CSI drivers, and how to install them, can be found on github.
- Documentation, including details around CSI limits can be found here.
- If the CSI version 2.0 driver is installed on K8s running on vSphere 6.7U3, the older CSI 1.0 driver features continue to work but the new CSI 2.0 features are not supported.
- If the CSI version 1.0.2 is installed on K8s running on vSphere 7.0, the CSI 1.0 driver features continue to work. CSI version 1.0.1 is not compatible with vSphere 7.
- CSI version 1.0.x and CSI version 2.0 on vSphere 6.7U3, vSphere with K8s, TKG ‘Guest’ and PKS 1.7 only support dynamically provisioned block volumes on vSphere storage.
- The dynamic creation of read-write-many (RWX) and read-only-many (ROX) file based Persistent Volumes is only available in vSphere 7.0 through vSAN 7.0 File Services.
- PV Encryption, offline volume grow and Topology/AZ support are only available on block based Persistent Volumes. These features are not available with PVs backed by file shares.
- Offline volume grow requires a minimum Kubernetes version of 1.16+, and is also a beta feature in this release.
- Multi-node Control Planes only supports having a single CSI Pod being active at any time (replica = 1 in controller manifest).
- WaitForConsumerFirst is a K8s feature that delays volume binding until the Pod has been scheduled, and is used for Pod and PV placement.
- Enterprise PKS/TGKI (v1.7) does not support vSphere 7.0 at the time of writing.
- [Update] CSI driver 2.0 does not yet support CSI snapshots (question was asked a number of times)