A closer look at vSphere with Kubernetes Permissions

In many of my recent posts about vSphere with Kubernetes, I use a single user (administrator@vsphere.local) to do all of my work. This allows me to carry out a range of activities without worrying about permissions. This vSphere Single Sign-On (SSO) administrator has “edit” permissions on all of the vK8s namespaces. In this post, I want to look at how to assign some different vSphere SSO users and permissions to different namespaces, and also how these permissions are implemented in the vK8s platform (through the Kubernetes ClusterRole and RoleBinding constructs).

Let’s start with a view of what a namespace looks like in vK8s. By default there are no permissions assigned to any namespaces.

Let’s now go ahead and create a new vSphere SSO user so I can use it for namespace permissions. The procedure to add  new user is found in the vSphere client under Administration > Single Sign On > Users and Groups.

Once the user is created, it can now be added with a role to the namespace permissions. There are two simple roles that can be assigned to a user, either view or edit.

For the purpose of this test, I gave user cormac a role that allows view in the namespace darren-ns, but edit in the namespace cormac-ns.

Let’s now login to vK8s as user cormac and see what we can and cannot do in the different namespaces.

$ kubectl-vsphere login  --server=https://20.0.0.1 --insecure-skip-tls-verify --vsphere-username \
cormac@vsphere.local

Password:********
Logged in successfully.

You have access to the following contexts:
   20.0.0.1
   cormac-ns
   darren-ns

If the context you wish to use is not in this list, you may need to try
logging in again later, or contact your cluster administrator.

To change context, use `kubectl config use-context <workload name>`

Now remember that user cormac can only view in the namespace darren-ns. Let’s switch to that namespace context and see what happens when we try to create a persistent volume claim in that namespace as user cormac.

$ kubectl config get-contexts
CURRENT   NAME               CLUSTER         AUTHINFO                                NAMESPACE
*         20.0.0.1           20.0.0.1        wcp:20.0.0.1:cormac@vsphere.local
          cormac-ns          20.0.0.1        wcp:20.0.0.1:cormac@vsphere.local       cormac-ns
          darren-ns          20.0.0.1        wcp:20.0.0.1:cormac@vsphere.local       darren-ns


$ kubectl config use-context darren-ns
Switched to context "darren-ns".


$ kubectl apply -f block-pvc.yaml
Error from server (Forbidden): error when creating "block-pvc.yaml": persistentvolumeclaims is forbidden: \
User "sso:cormac@vsphere.local" cannot create resource "persistentvolumeclaims" in API group "" in the \
namespace "darren-ns"

As we can see, user cormac is prevented from creating any new objects in that namespace as this user only has a view role in that namespace. Let’s try a similar operation in the namespace cormac-ns where user cormac has edit permissions.

$ kubectl config use-context cormac-ns
Switched to context "cormac-ns".

$ kubectl apply -f ./Pacific/BT-DEMO/pacific-vsan-block-rwo/block-pvc.yaml
persistentvolumeclaim/block-pvc created
$

This operation is successful. So how is it enforced? The first thing to be aware of is that when a user logs in to vK8s using the provided CLI command kubectl-vsphere login, the Kubernetes configuration (typically .kube/config) is updated with cluster, context and user details from the vSphere with Kubernetes environment. A Token Exchange Service runs on vCenter server when vK8s is enabled. This takes vSphere SSO credentials and converts them to JSON Web Tokens (JWT) for use with Kubernetes RBAC. This enables a vSphere user to login to the vK8s cluster. Entries similar to the following are added to the Kubernetes configuration file. We have the supervisor cluster and two namespace contexts , but all have the same user. In the user section of the configuration file, we can see the token.

contexts:
- context:
    cluster: 20.0.0.1
    user: wcp:20.0.0.1:cormac@vsphere.local
  name: 20.0.0.1

- context:
    cluster: 20.0.0.1
    namespace: cormac-ns
    user: wcp:20.0.0.1:cormac@vsphere.local
  name: cormac-ns

- context:
    cluster: 20.0.0.1
    namespace: darren-ns
    user: wcp:20.0.0.1:cormac@vsphere.local
  name: darren-ns


- name: wcp:20.0.0.1:cormac@vsphere.local
  user:
    token: eyJraWQiOiI4QTVCNUMxREExNDUwMTg3QzMyQjI1NDBDMDBCQ0YzMUYzMkU2NEE3IiwiYWxnIjoiUlMyNTYifQ....

The token is assigned on login, and lasts for 10 hours. Once it expires, the user will need to authenticate once more. On logout, the clusters, contexts and user token are removed from the configuration file.

But the question that now arises is how are the permissions seen earlier propagated from vSphere SSO to Kubernetes RBAC? How can this user have edit permissions on some namespaces, but view permissions on other namespace? To understand this, we need to look at Kubernetes constructs such as ClusterRole and RoleBinding (note: these can only be examined from the Supervisor Control Plane VMs, not from a vK8s namespace). Let’s first display the list of ClusterRoles. Here, among all the other roles, you can see that there are two roles, edit and view. These map back to the edit and view roles that we saw in the namespace permissions previously.

root@4216cd4619c10fbf96e48deaa2ed8228 [ ~ ]# kubectl get ClusterRole
NAME                                                                   AGE
admin                                                                  2d1h
aggregate-workload-cluster-admin                                       2d1h
cluster-admin                                                          2d1h
edit                                                                   2d1h
ncp-cluster-role                                                       2d1h
ncp-patch-role                                                         2d1h
nsx-node-agent-cluster-role                                            2d1h
system:aggregate-to-admin                                              2d1h
system:aggregate-to-edit                                               2d1h
system:aggregate-to-view                                               2d1h
system:auth-delegator                                                  2d1h
system:basic-user                                                      2d1h
system:certificates.k8s.io:certificatesigningrequests:nodeclient       2d1h
system:certificates.k8s.io:certificatesigningrequests:selfnodeclient   2d1h
system:controller:attachdetach-controller                              2d1h
.
.
.
system:persistent-volume-provisioner                                   2d1h
system:public-info-viewer                                              2d1h
system:volume-scheduler                                                2d1h
view                                                                   2d1h
vmware-image-agent-clusterrole                                         2d1h
vmware-image-controller-clusterrole                                    2d1h
vmware-registry-manager-clusterrole                                    2d1h
.
.
.
wcp:guest-clusters-view                                                2d1h
wcp:guest-clusters-view-cluster-scope-vmresources                      2d1h
wcp:schedext-role-annotations                                          2d1h
wcp:schedext-role-namespaces                                           2d1h
root@4216cd4619c10fbf96e48deaa2ed8228 [ ~ ]#

As one might imagine, the major difference between the two roles is that while the view role can typically do a get, list or watch on an object, the edit role can also do operations such as create, delete, patch and update objects. If you run a describe on the role, you can see the full set of capabilities associated with the role.

Let’s now take a look at the RoleBindings which can be thought of as controlling access to a namespace. We have assigned the user cormac to two namespaces, cormac-ns and darren-ns. We can see this user has a RoleBinding in each namespace.

root@4216cd4619c10fbf96e48deaa2ed8228 [ ~ ]# kubectl get RoleBinding -n cormac-ns
NAME                                               AGE
wcp:cormac-ns:group:vsphere.local:administrators   2d1h
wcp:cormac-ns:user:vsphere.local:administrator     89m
wcp:cormac-ns:user:vsphere.local:cormac            93m

root@4216cd4619c10fbf96e48deaa2ed8228 [ ~ ]# kubectl get RoleBinding -n darren-ns
NAME                                               AGE
wcp:darren-ns:group:vsphere.local:administrators   25h
wcp:darren-ns:user:vsphere.local:administrator     89m
wcp:darren-ns:user:vsphere.local:cormac            90m

Let’s now examine the RoleBindings in more detail, and one would assume that the RoleBinding for cormac in the namespace cormac-ns has an edit ClusterRole, whilst the RoleBinding in the namespace darren-ns only has a view ClusterRole. That is indeed the case. Let’s look at cormac-ns first:

root@4216cd4619c10fbf96e48deaa2ed8228 [ ~ ]# kubectl describe RoleBinding wcp:cormac-ns:user:vsphere.local:cormac -n cormac-ns

Name:         wcp:cormac-ns:user:vsphere.local:cormac
Labels:       managedBy=vSphere
Annotations:  <none>
Role:
  Kind:  ClusterRole
  Name:  edit
Subjects:
  Kind  Name                      Namespace
  ----  ----                      ---------
  User  sso:cormac@vsphere.local


root@4216cd4619c10fbf96e48deaa2ed8228 [ ~ ]# kubectl get RoleBinding wcp:cormac-ns:user:vsphere.local:cormac -n cormac-ns -o yaml

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  creationTimestamp: "2020-07-08T10:47:53Z"
  labels:
    managedBy: vSphere
  name: wcp:cormac-ns:user:vsphere.local:cormac
  namespace: cormac-ns
  resourceVersion: "1232328"
  selfLink: /apis/rbac.authorization.k8s.io/v1/namespaces/cormac-ns/rolebindings/wcp:cormac-ns:user:vsphere.local:cormac
  uid: eec05c61-57a5-40ba-814c-7ade94c34011
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: edit
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: User
  name: sso:cormac@vsphere.local
root@4216cd4619c10fbf96e48deaa2ed8228 [ ~ ]#

We can clearly see the edit ClusterRole in the cormac-ns RoleBinding for use cormac. Let’s look at darren-ns next:

root@4216cd4619c10fbf96e48deaa2ed8228 [ ~ ]# kubectl describe rolebinding wcp:darren-ns:user:vsphere.local:cormac -n darren-ns

Name:         wcp:darren-ns:user:vsphere.local:cormac
Labels:       managedBy=vSphere
Annotations:  <none>
Role:
  Kind:  ClusterRole
  Name:  view
Subjects:
  Kind  Name                      Namespace
  ----  ----                      ---------
  User  sso:cormac@vsphere.local


root@4216cd4619c10fbf96e48deaa2ed8228 [ ~ ]# kubectl get rolebinding wcp:darren-ns:user:vsphere.local:cormac -n darren-ns -o yaml

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  creationTimestamp: "2020-07-08T10:50:18Z"
  labels:
    managedBy: vSphere
  name: wcp:darren-ns:user:vsphere.local:cormac
  namespace: darren-ns
  resourceVersion: "1233346"
  selfLink: /apis/rbac.authorization.k8s.io/v1/namespaces/darren-ns/rolebindings/wcp:darren-ns:user:vsphere.local:cormac
  uid: ac47c339-2503-40f6-8726-1b00ea243485
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: view
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: User
  name: sso:cormac@vsphere.local

And finally we can see that the ClusterRole used for the cormac user in the darren-ns namespace is view. This is why this user could not create a PVC earlier. And with that, I hope that has given you a good insight into how vSphere SSO logins are implemented in vSphere with Kubernetes using ClusterRole and RoleBindings.