Using Tanzu Mission Control for managing LDAP/AD access policies for workload clusters

Tanzu Mission ControlI’ve recently been looking at some of the features around Tanzu Mission Control. Tanzu Mission Control (or TMC for short) is a VMware SaaS offering for managing and monitoring your Kubernetes Clusters across multiple clouds. My particular interest on this occasion was around the access policy features, especially when the Tanzu Kubernetes Grid (TKG) workload clusters were deployed with LDAP/Active Directory integration via the Pinniped and Dex packages that are available with TKG. In this post, I will rollout my TKG management cluster, followed by a pair of TKG workload clusters. The TKG management cluster will be automatically integrated with TMC at deployment time. Once the workload clusters are deployed, I will use TMC to manage them, and bring them under TMC control. I will then create a cluster group in TMC that contains both of my workload clusters and create my LDAP/AD access policy / ClusterRoleBinding at the cluster group level. I should then observe that my authenticated LDAP/AD non-admin user should be able to successfully login to both workload clusters. As a final step, I will deploy a third TKG workload cluster directly from Tanzu Mission Control. This third workload cluster will be provisioned as part of the same cluster group, so it will automatically inherit the Access Policy /ClusterRoleBinding set at the cluster group level. Thus, my LDAP/AD non-admin user should then be able to access the third workload cluster immediately after it is provisioned. Let’s go through the deployment steps.

Step 1. Deploy the TKG Management Cluster

This is the manifest used to deploy the TKG (v1.3.1) Management Cluster. It has the following integration points.

  • Integrated with NSX ALB (Avi Vantage Load Balancer)
  • Integrated with Active Directory / LDAP
  • Integrated with Tanzu Mission Control (TMC)

The cluster can be deployed via the UI, or via the command line. The UI approach also creates a configuration file which can be used to deploy additional management clusters, or indeed to deploy the same management cluster. This is the configuration file that was used to deploy the management cluster, with some information obfuscated.

AVI_CA_DATA_B64: LS0t...LS0tLQo=
AVI_CLOUD_NAME: Default-Cloud
AVI_CONTROLLER: xx.yy.25.165
AVI_DATA_NETWORK: VL574-DPortGroup
AVI_DATA_NETWORK_CIDR: xx.yy.112.128/26
AVI_ENABLE: "true"
AVI_LABELS: ""
AVI_PASSWORD: <encoded:Vk13YXJlMTIzIQ==>
AVI_SERVICE_ENGINE_GROUP: Default-Group
AVI_USERNAME: admin
CLUSTER_CIDR: 100.96.0.0/11
CLUSTER_NAME: mgmt-cjh
CLUSTER_PLAN: prod
ENABLE_CEIP_PARTICIPATION: "true"
ENABLE_MHC: "true"
IDENTITY_MANAGEMENT_TYPE: ldap
INFRASTRUCTURE_PROVIDER: vsphere
LDAP_BIND_DN: cn=Administrator,cn=Users,dc=rainpole,dc=com
LDAP_BIND_PASSWORD: <encoded:VnhSYWlsITIz>
LDAP_GROUP_SEARCH_BASE_DN: dc=rainpole,dc=com
LDAP_GROUP_SEARCH_FILTER: (objectClass=group)
LDAP_GROUP_SEARCH_GROUP_ATTRIBUTE: ""
LDAP_GROUP_SEARCH_NAME_ATTRIBUTE: ""
LDAP_GROUP_SEARCH_USER_ATTRIBUTE: ""
LDAP_HOST: dc01.rainpole.com:636
LDAP_ROOT_CA_DATA_B64: LS0t...LQ==
LDAP_USER_SEARCH_BASE_DN: cn=Users,dc=rainpole,dc=com
LDAP_USER_SEARCH_FILTER: ""
LDAP_USER_SEARCH_NAME_ATTRIBUTE: userPrincipalName
LDAP_USER_SEARCH_USERNAME: userPrincipalName
OIDC_IDENTITY_PROVIDER_CLIENT_ID: ""
OIDC_IDENTITY_PROVIDER_CLIENT_SECRET: ""
OIDC_IDENTITY_PROVIDER_GROUPS_CLAIM: ""
OIDC_IDENTITY_PROVIDER_ISSUER_URL: ""
OIDC_IDENTITY_PROVIDER_NAME: ""
OIDC_IDENTITY_PROVIDER_SCOPES: ""
OIDC_IDENTITY_PROVIDER_USERNAME_CLAIM: ""
SERVICE_CIDR: 100.64.0.0/13
TKG_HTTP_PROXY_ENABLED: "false"
TMC_REGISTRATION_URL: https://zzz.tmc.cloud.vmware.com/installer?id=xxx&source=registration&type=tkgm
VSPHERE_CONTROL_PLANE_DISK_GIB: "40"
VSPHERE_CONTROL_PLANE_ENDPOINT: xx.yy.25.245
VSPHERE_CONTROL_PLANE_MEM_MIB: "8192"
VSPHERE_CONTROL_PLANE_NUM_CPUS: "2"
VSPHERE_DATACENTER: /CH-OCTO-DC
VSPHERE_DATASTORE: /CH-OCTO-DC/datastore/vsanDatastore
VSPHERE_FOLDER: /CH-OCTO-DC/vm/TKG
VSPHERE_NETWORK: VL530-DPortGroup
VSPHERE_PASSWORD: <encoded:QWRtaW4hMjM=>
VSPHERE_RESOURCE_POOL: /CH-OCTO-DC/host/CH-Cluster/Resources
VSPHERE_SERVER: vcsa-cormac.eng.vmware.com
VSPHERE_SSH_AUTHORIZED_KEY: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC/IxQRUbSOKcRdzDqdNYgZ5T2xG4utZHRw68q4KpAW1VMyFagdS9voKIdv5S+Q5Ndf+6w5RawdlMdYgiwSxZahAlUupTsXrmJkts9AVFJqkqRERQRMAsk/qgc9Nxju++yECfosXtsQtK2OhK7BnjhQ8wqNTlKduPfFA3Tk06I/2CbstKQ/G3rd1U5k7FkIcxpUqKN93ebIPneCrML2jZc8IAjrcNSRA2uHZFa6ObkBjtXZmQ2J1d1jm1sfEjexzFdQkEHfoGcMiNl5BNKARMjx/YMSCzQmnWo683SYjVSbPAuGvUY3AkHItbBqMiRsZgbYPmwOnWBW3NJGMtU8Ylc+bg1k/ENveHlwGBu8InSJ7YjDjoH0DKC0gY9NiFZ3fPq4xrXgirFKQWv1NVUC3atmyofTNQcma3EQR/n7448MvAnegq+J8Dcv2rz1c2dTox9oD8yEzMGrfOSUo3NXc3ypkEjkNETg9JNG632TkKIEH+Qdkbdce8vjzgVfVb9c99ZcwlqfLTojwW2Je4rhwpU+RJ3cAuixIfDxoYEW096QG7MSYGJbqKG4lW7vmrbL+WNmpfHIyssMHzYpzRFzyrotNi50TIdpYNzgXlIVkkmZcxSaTenwTTjP2NsZ6LczThZTXNcGDNJ7dsFib5O4KO+oOQ4RARGKXrWHKxZjiNBVdQ== cormac
VSPHERE_TLS_THUMBPRINT: 56:xx:C5:47:xx:00:C0:yy:97:yy:A5:14:zz:0E:zz:65:ww:CF:48:90
VSPHERE_USERNAME: administrator@vsphere.local
VSPHERE_WORKER_DISK_GIB: "40"
VSPHERE_WORKER_MEM_MIB: "8192"
VSPHERE_WORKER_NUM_CPUS: "2"

Step 2. Deploy TKG Workload Clusters

This is the manifest used to deploy the TKG (v1.3.1) Workload Clusters. You will notice that this also has the following features configured.

  • Integrated with NSX ALB (Avi Vantage Load Balancer)
  • Integrated with Active Directory / LDAP

This will automatically appear as workload cluster in Tanzu Mission Control, and from there you can decide if you want to manage it via TMC. This is discussed in step 3.

AVI_CA_DATA_B64: LS0tL...tLQo=
AVI_CLOUD_NAME: Default-Cloud
AVI_CONTROLLER: 10.202.25.165
AVI_DATA_NETWORK: VL574-DPortGroup
AVI_DATA_NETWORK_CIDR: xx.yy.112.128/26
AVI_ENABLE: "true"
AVI_LABELS: ""
AVI_PASSWORD: <encoded:Vk13YXJlMTIzIQ==>
AVI_SERVICE_ENGINE_GROUP: Default-Group
AVI_USERNAME: admin
CLUSTER_CIDR: 100.96.1.0/11
CLUSTER_NAME: workload1
CLUSTER_PLAN: dev
ENABLE_CEIP_PARTICIPATION: "true"
ENABLE_MHC: "true"
IDENTITY_MANAGEMENT_TYPE: ldap
INFRASTRUCTURE_PROVIDER: vsphere
SERVICE_CIDR: 100.64.1.0/13
TKG_HTTP_PROXY_ENABLED: "false"
VSPHERE_CONTROL_PLANE_DISK_GIB: "40"
VSPHERE_CONTROL_PLANE_ENDPOINT: xx.yy.25.244
VSPHERE_CONTROL_PLANE_MEM_MIB: "8192"
VSPHERE_CONTROL_PLANE_NUM_CPUS: "2"
VSPHERE_DATACENTER: /CH-OCTO-DC
VSPHERE_DATASTORE: /CH-OCTO-DC/datastore/vsanDatastore
VSPHERE_FOLDER: /CH-OCTO-DC/vm/TKG
VSPHERE_NETWORK: VL530-DPortGroup
VSPHERE_PASSWORD: <encoded:QWRtaW4hMjM=>
VSPHERE_RESOURCE_POOL: /CH-OCTO-DC/host/CH-Cluster/Resources
VSPHERE_SERVER: vcsa-cormac.eng.vmware.com
VSPHERE_SSH_AUTHORIZED_KEY: ssh-rsa AAAAB...dQ== cormac
VSPHERE_TLS_THUMBPRINT: 56:xx:C5:47:yy:00:zz:68:ww:FF:xx:14:6Byy0E:37:zz:3C:xx:48:90
VSPHERE_USERNAME: administrator@vsphere.local
VSPHERE_WORKER_DISK_GIB: "40"
VSPHERE_WORKER_MEM_MIB: "8192"
VSPHERE_WORKER_NUM_CPUS: "2"

Let’s take a look at Tanzu Mission Control next.

Step 3 : Tanzu Mission Control Integration

The TKG management cluster appears in TMC automatically due to the integration specification in the management cluster deployment manifest. This is what it looks like:

TMC TKG Management Cluster

Once the worklaod clusters are deployed, they will appear under the Workload clusters view in Administration > Management Clusters > ‘Your Management Cluster’. Simply select the Workload Clusters and click on the Manage Workload button to bring them under the control of TMC. This creates a new vmware-system-tmc namespace on the workload cluster, and deployed the necessary TMC agents to the cluster.

TMC TKG Workload Cluster

Step 4: Active Directory / LDAP integration on workload cluster

This is the crux of the issue that is being investigated in this post. In a non-TMC managed workload cluster, the following steps would be followed to enable an end user / developer to access the cluster. These are described in much more detail in this blog post.

  1. Switch to non-admin context
  2. Try to query the cluster – (% kubectl get nodes) – Dex portal launches in browser
  3. Provide AD/LDAP credential for end-user in Dex portal
  4. Observe `Error from server (Forbidden): nodes is forbidden: User “chogan@rainpole.com” cannot list resource “nodes” in API group “” at the cluster scope`
  5. Switch to admin context
  6. Create and Apply ClusterRoleBinding manifest for user, specifying cluster privileges/role
  7. Switch back to non-admin context
  8. Try to query the cluster – (% kubectl get nodes) – now it should succeed

Now imagine a Kubernetes cluster admin doing these steps for every user on every cluster. Pretty painful! TMC provides a way to automate this process through it’s access policies. Let’s look at that next.

Step 5: Access Policy in TMC

Through Policy > Access in Tanzu Mission Control we can very quickly set a ClusterRoleBinding policy that can be applied at the individual cluster level, to clusters groups, or to the whole of the organization (all clusters). In the screenshot below, a ClusterRoleBinding for an LDAP/AD user called chogan@rainpole.com has been added to a single cluster under the default cluster group.

TMC Policy Access at the Cluster Level

But as mentioned, this policy can be applied to a group of clusters as well. In this example, I have grouped two workload clusters into a cluster group, and applied that policy once at the cluster group level. All workload clusters in the group will automatically get the ClusterRoleBinding.

TMC Policy Access at the Cluster Group Level

And if you had an access policy that you would like to apply for every cluster, simply apply the policy at the Organization level. As you can see, using TMC make Access Control so much easier when dealing with many disparate Kubernetes clusters. Using TMC, the steps are simplified as follows.

  1. Navigate to TMC
  2. Under Policy > Access, select Clusters, depending on scope of access you wish to give to the user.
  3. If individual cluster, select individual cluster and then expand it under Direct Access Policies. If cluster group, select the cluster group and again expand it under Direct Access Policies. If Organization, select that level in the tree and expand once again to view Direct Access Policies.
  4. Select Cluster Role, e.g cluster.admin
  5. Select Identity i.e. User or Group
  6. Select User identity, e.g. chogan@rainpole.com
  7. Switch to command line and from non-admin context, try to query the cluster – (% kubectl get nodes)
  8. Provide AD/LDAP credential for end-user in Dex portal (pops up in a browser window)
  9. The cluster query command should succeed immediately.

Now imagine that you have many, many K8s clusters where this developer/admin/user needs access. Simply change the scope of the ClusterRoleBinding to be organization or cluster group level, and all clusters in that tree inherit the policy. That is very cool. A final point on Access Policies in TMC is that it is possible to have TMC’s Identity Management and external LDAP/Active Directory both be active on a workload cluster.

Step 6 : Deploy new workload cluster from TMC

I had 2 outstanding questions to answer with TMC:

  1. Does the new workload cluster, deployed from TMC, also get AD/LDAP integration if the management cluster has it configured? The answer is YES.
  2. Does the new cluster automatically inherit Access Policies if they are configured further up the tree? i.e. if the cluster added to a cluster group while it is being provisioned? The answer is also YES.

Here is a third workload cluster provisioned directly from TMC in the same cluster group as the two previous workload clusters. As you can see, it has automatically received the AD/LDAP user from the cluster group, meaning that this user can automatically login to the cluster.

TMC Access Policy inherited by workload cluster deployed by TMC

Another way to confirm that the configuration features associated with the management cluster (e.g. NSX ALB integration, LDAP integration) have been added to the workload cluster is to check the pod listing on this newly deployed workload cluster.

$ kubectl get pods -A
NAMESPACE            NAME                                                          READY  STATUS      RESTARTS  AGE
avi-system           ako-0                                                          1/1    Running    0          16h
kube-system          antrea-agent-wtmvb                                             2/2    Running    0          16h
kube-system          antrea-agent-zngvx                                             2/2    Running    0          16h
kube-system          antrea-controller-556dc8489c-zl47z                             1/1    Running    0          16h
kube-system          coredns-68d49685bd-46wq6                                       1/1    Running    0          16h
kube-system          coredns-68d49685bd-nsrpm                                       1/1    Running    0          16h
kube-system          etcd-workload3a-control-plane-28n6c                            1/1    Running    0          16h
kube-system          kube-apiserver-workload3a-control-plane-28n6c                  1/1    Running    0          16h
kube-system          kube-controller-manager-workload3a-control-plane-28n6c         1/1    Running    0          16h
kube-system          kube-proxy-lfmc7                                               1/1    Running    0          16h
kube-system          kube-proxy-tvsbc                                               1/1    Running    0          16h
kube-system          kube-scheduler-workload3a-control-plane-28n6c                  1/1    Running    0          16h
kube-system          kube-vip-workload3a-control-plane-28n6c                        1/1    Running    0          16h
kube-system          metrics-server-79476747c9-g6rmf                                1/1    Running    0          16h
kube-system          vsphere-cloud-controller-manager-xsdcj                         1/1    Running    0          16h
kube-system          vsphere-csi-controller-fd4b88d9b-j6rpm                         6/6    Running    0          16h
kube-system          vsphere-csi-node-jbzpl                                         3/3    Running    0          16h
kube-system          vsphere-csi-node-kjzpc                                         3/3    Running    0          16h
pinniped-concierge   pinniped-concierge-588c757cf4-2qbpv                            1/1    Running    0          16h
pinniped-concierge   pinniped-concierge-588c757cf4-56k4s                            1/1    Running    0          16h
pinniped-concierge   pinniped-concierge-kube-cert-agent-d51497f6                    1/1    Running    0          16h
pinniped-supervisor  pinniped-post-deploy-job-ckxqt                                 0/1    Error      0          16h
pinniped-supervisor  pinniped-post-deploy-job-xv2zr                                 0/1    Completed  0          16h
tkg-system           kapp-controller-5b4756cfb-84hj2                                1/1    Running    0          16h
vmware-system-tmc    agent-updater-59b85d846d-7zftx                                 1/1    Running    0          16h
vmware-system-tmc    agentupdater-workload-1633592640-b68lz                         0/1    Completed  0          43s
vmware-system-tmc    cluster-auth-pinniped-6b99b89b6f-bpff8                         1/1    Running    0          16h
vmware-system-tmc    cluster-auth-pinniped-6b99b89b6f-hxwzx                         1/1    Running    0          16h
vmware-system-tmc    cluster-auth-pinniped-kube-cert-agent-79bbdd7c44-49nqx         1/1    Running    0          16h
vmware-system-tmc    cluster-health-extension-755c5bf45d-kwccn                      1/1    Running    0          16h
vmware-system-tmc    extension-manager-f78b7776d-6s984                              1/1    Running    0          16h
vmware-system-tmc    extension-updater-5bd46745c7-gb6cc                             1/1    Running    0          16h
vmware-system-tmc    gatekeeper-operator-manager-9bf45f5f5-r7lhf                    1/1    Running    0          16h
vmware-system-tmc    inspection-extension-5c7567d669-7b77t                          1/1    Running    0          16h
vmware-system-tmc    intent-agent-77d75dd59f-jb6xp                                  1/1    Running    0          16h
vmware-system-tmc    logs-collector-cluster-auth-pinniped-20211006150814-pngks      0/1    Completed  0          16h
vmware-system-tmc    logs-collector-cluster-health-extension-20211006150814-k59zt   0/1    Completed  0          16h
vmware-system-tmc    logs-collector-extension-manager-20211006150815-g2ghh          0/1    Completed  0          16h
vmware-system-tmc    logs-collector-gatekeeper-operator-20211006150814-wmq5b        0/1    Completed  0          16h
vmware-system-tmc    logs-collector-inspection-20211006150814-s8t8k                 0/1    Completed  0          16h
vmware-system-tmc    logs-collector-intent-agent-20211006150814-cx4mn               0/1    Completed  0          16h
vmware-system-tmc    logs-collector-policy-insight-extension-20211006150814-jcdgz   0/1    Completed  0          16h
vmware-system-tmc    logs-collector-policy-sync-extension-20211006150814-2f9r5      0/1    Completed  0          16h
vmware-system-tmc    logs-collector-tmc-observer-20211006150814-xhvnh               0/1    Completed  0          16h
vmware-system-tmc    policy-insight-extension-manager-5858b4dc68-l2jbx              1/1    Running    0          16h
vmware-system-tmc    policy-sync-extension-cfcd4b7b5-4lh64                          1/1    Running    0          16h
vmware-system-tmc    sync-agent-f968b579-bq244                                      1/1    Running    0          16h
vmware-system-tmc    tmc-observer-67cb4c7fc8-7tq9d                                  1/1    Running    0          16h

The ako-0 pod in the avi-system namespace provides the NSX ALB integration. The pods in the pinniped-concierge namespace are for the LDAP/AD integration. This looks good! Hopefully you can now appreciate how good this functionality would be for a Kubernetes cluster administrator.  It certainly makes access policy management a lot easier when trying to manage the roles of multiple users across multiple Kubernetes clusters.

If you are interested in Tanzu Mission Control, why not check out the newly announced Tanzu Mission Control Starter, a free tier of Tanzu Mission Control. This was just announced at VMworld 2021 along with Tanzu Community Edition. Check it out!