I’ve recently been looking at some of the features around Tanzu Mission Control. Tanzu Mission Control (or TMC for short) is a VMware SaaS offering for managing and monitoring your Kubernetes Clusters across multiple clouds. My particular interest on this occasion was around the access policy features, especially when the Tanzu Kubernetes Grid (TKG) workload clusters were deployed with LDAP/Active Directory integration via the Pinniped and Dex packages that are available with TKG. In this post, I will rollout my TKG management cluster, followed by a pair of TKG workload clusters. The TKG management cluster will be automatically integrated with TMC at deployment time. Once the workload clusters are deployed, I will use TMC to manage them, and bring them under TMC control. I will then create a cluster group in TMC that contains both of my workload clusters and create my LDAP/AD access policy / ClusterRoleBinding at the cluster group level. I should then observe that my authenticated LDAP/AD non-admin user should be able to successfully login to both workload clusters. As a final step, I will deploy a third TKG workload cluster directly from Tanzu Mission Control. This third workload cluster will be provisioned as part of the same cluster group, so it will automatically inherit the Access Policy /ClusterRoleBinding set at the cluster group level. Thus, my LDAP/AD non-admin user should then be able to access the third workload cluster immediately after it is provisioned. Let’s go through the deployment steps.
Step 1. Deploy the TKG Management Cluster
This is the manifest used to deploy the TKG (v1.3.1) Management Cluster. It has the following integration points.
- Integrated with NSX ALB (Avi Vantage Load Balancer)
- Integrated with Active Directory / LDAP
- Integrated with Tanzu Mission Control (TMC)
The cluster can be deployed via the UI, or via the command line. The UI approach also creates a configuration file which can be used to deploy additional management clusters, or indeed to deploy the same management cluster. This is the configuration file that was used to deploy the management cluster, with some information obfuscated.
AVI_CA_DATA_B64: LS0t...LS0tLQo= AVI_CLOUD_NAME: Default-Cloud AVI_CONTROLLER: xx.yy.25.165 AVI_DATA_NETWORK: VL574-DPortGroup AVI_DATA_NETWORK_CIDR: xx.yy.112.128/26 AVI_ENABLE: "true" AVI_LABELS: "" AVI_PASSWORD: <encoded:Vk13YXJlMTIzIQ==> AVI_SERVICE_ENGINE_GROUP: Default-Group AVI_USERNAME: admin CLUSTER_CIDR: 100.96.0.0/11 CLUSTER_NAME: mgmt-cjh CLUSTER_PLAN: prod ENABLE_CEIP_PARTICIPATION: "true" ENABLE_MHC: "true" IDENTITY_MANAGEMENT_TYPE: ldap INFRASTRUCTURE_PROVIDER: vsphere LDAP_BIND_DN: cn=Administrator,cn=Users,dc=rainpole,dc=com LDAP_BIND_PASSWORD: <encoded:VnhSYWlsITIz> LDAP_GROUP_SEARCH_BASE_DN: dc=rainpole,dc=com LDAP_GROUP_SEARCH_FILTER: (objectClass=group) LDAP_GROUP_SEARCH_GROUP_ATTRIBUTE: "" LDAP_GROUP_SEARCH_NAME_ATTRIBUTE: "" LDAP_GROUP_SEARCH_USER_ATTRIBUTE: "" LDAP_HOST: dc01.rainpole.com:636 LDAP_ROOT_CA_DATA_B64: LS0t...LQ== LDAP_USER_SEARCH_BASE_DN: cn=Users,dc=rainpole,dc=com LDAP_USER_SEARCH_FILTER: "" LDAP_USER_SEARCH_NAME_ATTRIBUTE: userPrincipalName LDAP_USER_SEARCH_USERNAME: userPrincipalName OIDC_IDENTITY_PROVIDER_CLIENT_ID: "" OIDC_IDENTITY_PROVIDER_CLIENT_SECRET: "" OIDC_IDENTITY_PROVIDER_GROUPS_CLAIM: "" OIDC_IDENTITY_PROVIDER_ISSUER_URL: "" OIDC_IDENTITY_PROVIDER_NAME: "" OIDC_IDENTITY_PROVIDER_SCOPES: "" OIDC_IDENTITY_PROVIDER_USERNAME_CLAIM: "" SERVICE_CIDR: 100.64.0.0/13 TKG_HTTP_PROXY_ENABLED: "false" TMC_REGISTRATION_URL: https://zzz.tmc.cloud.vmware.com/installer?id=xxx&source=registration&type=tkgm VSPHERE_CONTROL_PLANE_DISK_GIB: "40" VSPHERE_CONTROL_PLANE_ENDPOINT: xx.yy.25.245 VSPHERE_CONTROL_PLANE_MEM_MIB: "8192" VSPHERE_CONTROL_PLANE_NUM_CPUS: "2" VSPHERE_DATACENTER: /CH-OCTO-DC VSPHERE_DATASTORE: /CH-OCTO-DC/datastore/vsanDatastore VSPHERE_FOLDER: /CH-OCTO-DC/vm/TKG VSPHERE_NETWORK: VL530-DPortGroup VSPHERE_PASSWORD: <encoded:QWRtaW4hMjM=> VSPHERE_RESOURCE_POOL: /CH-OCTO-DC/host/CH-Cluster/Resources VSPHERE_SERVER: vcsa-cormac.eng.vmware.com VSPHERE_SSH_AUTHORIZED_KEY: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC/IxQRUbSOKcRdzDqdNYgZ5T2xG4utZHRw68q4KpAW1VMyFagdS9voKIdv5S+Q5Ndf+6w5RawdlMdYgiwSxZahAlUupTsXrmJkts9AVFJqkqRERQRMAsk/qgc9Nxju++yECfosXtsQtK2OhK7BnjhQ8wqNTlKduPfFA3Tk06I/2CbstKQ/G3rd1U5k7FkIcxpUqKN93ebIPneCrML2jZc8IAjrcNSRA2uHZFa6ObkBjtXZmQ2J1d1jm1sfEjexzFdQkEHfoGcMiNl5BNKARMjx/YMSCzQmnWo683SYjVSbPAuGvUY3AkHItbBqMiRsZgbYPmwOnWBW3NJGMtU8Ylc+bg1k/ENveHlwGBu8InSJ7YjDjoH0DKC0gY9NiFZ3fPq4xrXgirFKQWv1NVUC3atmyofTNQcma3EQR/n7448MvAnegq+J8Dcv2rz1c2dTox9oD8yEzMGrfOSUo3NXc3ypkEjkNETg9JNG632TkKIEH+Qdkbdce8vjzgVfVb9c99ZcwlqfLTojwW2Je4rhwpU+RJ3cAuixIfDxoYEW096QG7MSYGJbqKG4lW7vmrbL+WNmpfHIyssMHzYpzRFzyrotNi50TIdpYNzgXlIVkkmZcxSaTenwTTjP2NsZ6LczThZTXNcGDNJ7dsFib5O4KO+oOQ4RARGKXrWHKxZjiNBVdQ== cormac VSPHERE_TLS_THUMBPRINT: 56:xx:C5:47:xx:00:C0:yy:97:yy:A5:14:zz:0E:zz:65:ww:CF:48:90 VSPHERE_USERNAME: firstname.lastname@example.org VSPHERE_WORKER_DISK_GIB: "40" VSPHERE_WORKER_MEM_MIB: "8192" VSPHERE_WORKER_NUM_CPUS: "2"
Step 2. Deploy TKG Workload Clusters
This is the manifest used to deploy the TKG (v1.3.1) Workload Clusters. You will notice that this also has the following features configured.
- Integrated with NSX ALB (Avi Vantage Load Balancer)
- Integrated with Active Directory / LDAP
This will automatically appear as workload cluster in Tanzu Mission Control, and from there you can decide if you want to manage it via TMC. This is discussed in step 3.
AVI_CA_DATA_B64: LS0tL...tLQo= AVI_CLOUD_NAME: Default-Cloud AVI_CONTROLLER: 10.202.25.165 AVI_DATA_NETWORK: VL574-DPortGroup AVI_DATA_NETWORK_CIDR: xx.yy.112.128/26 AVI_ENABLE: "true" AVI_LABELS: "" AVI_PASSWORD: <encoded:Vk13YXJlMTIzIQ==> AVI_SERVICE_ENGINE_GROUP: Default-Group AVI_USERNAME: admin CLUSTER_CIDR: 100.96.1.0/11 CLUSTER_NAME: workload1 CLUSTER_PLAN: dev ENABLE_CEIP_PARTICIPATION: "true" ENABLE_MHC: "true" IDENTITY_MANAGEMENT_TYPE: ldap INFRASTRUCTURE_PROVIDER: vsphere SERVICE_CIDR: 100.64.1.0/13 TKG_HTTP_PROXY_ENABLED: "false" VSPHERE_CONTROL_PLANE_DISK_GIB: "40" VSPHERE_CONTROL_PLANE_ENDPOINT: xx.yy.25.244 VSPHERE_CONTROL_PLANE_MEM_MIB: "8192" VSPHERE_CONTROL_PLANE_NUM_CPUS: "2" VSPHERE_DATACENTER: /CH-OCTO-DC VSPHERE_DATASTORE: /CH-OCTO-DC/datastore/vsanDatastore VSPHERE_FOLDER: /CH-OCTO-DC/vm/TKG VSPHERE_NETWORK: VL530-DPortGroup VSPHERE_PASSWORD: <encoded:QWRtaW4hMjM=> VSPHERE_RESOURCE_POOL: /CH-OCTO-DC/host/CH-Cluster/Resources VSPHERE_SERVER: vcsa-cormac.eng.vmware.com VSPHERE_SSH_AUTHORIZED_KEY: ssh-rsa AAAAB...dQ== cormac VSPHERE_TLS_THUMBPRINT: 56:xx:C5:47:yy:00:zz:68:ww:FF:xx:14:6Byy0E:37:zz:3C:xx:48:90 VSPHERE_USERNAME: email@example.com VSPHERE_WORKER_DISK_GIB: "40" VSPHERE_WORKER_MEM_MIB: "8192" VSPHERE_WORKER_NUM_CPUS: "2"
Let’s take a look at Tanzu Mission Control next.
Step 3 : Tanzu Mission Control Integration
The TKG management cluster appears in TMC automatically due to the integration specification in the management cluster deployment manifest. This is what it looks like:
Once the worklaod clusters are deployed, they will appear under the Workload clusters view in Administration > Management Clusters > ‘Your Management Cluster’. Simply select the Workload Clusters and click on the Manage Workload button to bring them under the control of TMC. This creates a new vmware-system-tmc namespace on the workload cluster, and deployed the necessary TMC agents to the cluster.
Step 4: Active Directory / LDAP integration on workload cluster
This is the crux of the issue that is being investigated in this post. In a non-TMC managed workload cluster, the following steps would be followed to enable an end user / developer to access the cluster. These are described in much more detail in this blog post.
- Switch to non-admin context
- Try to query the cluster – (% kubectl get nodes) – Dex portal launches in browser
- Provide AD/LDAP credential for end-user in Dex portal
- Observe `Error from server (Forbidden): nodes is forbidden: User “firstname.lastname@example.org” cannot list resource “nodes” in API group “” at the cluster scope`
- Switch to admin context
- Create and Apply ClusterRoleBinding manifest for user, specifying cluster privileges/role
- Switch back to non-admin context
- Try to query the cluster – (% kubectl get nodes) – now it should succeed
Now imagine a Kubernetes cluster admin doing these steps for every user on every cluster. Pretty painful! TMC provides a way to automate this process through it’s access policies. Let’s look at that next.
Step 5: Access Policy in TMC
Through Policy > Access in Tanzu Mission Control we can very quickly set a ClusterRoleBinding policy that can be applied at the individual cluster level, to clusters groups, or to the whole of the organization (all clusters). In the screenshot below, a ClusterRoleBinding for an LDAP/AD user called email@example.com has been added to a single cluster under the default cluster group.
But as mentioned, this policy can be applied to a group of clusters as well. In this example, I have grouped two workload clusters into a cluster group, and applied that policy once at the cluster group level. All workload clusters in the group will automatically get the ClusterRoleBinding.
And if you had an access policy that you would like to apply for every cluster, simply apply the policy at the Organization level. As you can see, using TMC make Access Control so much easier when dealing with many disparate Kubernetes clusters. Using TMC, the steps are simplified as follows.
- Navigate to TMC
- Under Policy > Access, select Clusters, depending on scope of access you wish to give to the user.
- If individual cluster, select individual cluster and then expand it under Direct Access Policies. If cluster group, select the cluster group and again expand it under Direct Access Policies. If Organization, select that level in the tree and expand once again to view Direct Access Policies.
- Select Cluster Role, e.g cluster.admin
- Select Identity i.e. User or Group
- Select User identity, e.g. firstname.lastname@example.org
- Switch to command line and from non-admin context, try to query the cluster – (% kubectl get nodes)
- Provide AD/LDAP credential for end-user in Dex portal (pops up in a browser window)
- The cluster query command should succeed immediately.
Now imagine that you have many, many K8s clusters where this developer/admin/user needs access. Simply change the scope of the ClusterRoleBinding to be organization or cluster group level, and all clusters in that tree inherit the policy. That is very cool. A final point on Access Policies in TMC is that it is possible to have TMC’s Identity Management and external LDAP/Active Directory both be active on a workload cluster.
Step 6 : Deploy new workload cluster from TMC
I had 2 outstanding questions to answer with TMC:
- Does the new workload cluster, deployed from TMC, also get AD/LDAP integration if the management cluster has it configured? The answer is YES.
- Does the new cluster automatically inherit Access Policies if they are configured further up the tree? i.e. if the cluster added to a cluster group while it is being provisioned? The answer is also YES.
Here is a third workload cluster provisioned directly from TMC in the same cluster group as the two previous workload clusters. As you can see, it has automatically received the AD/LDAP user from the cluster group, meaning that this user can automatically login to the cluster.
Another way to confirm that the configuration features associated with the management cluster (e.g. NSX ALB integration, LDAP integration) have been added to the workload cluster is to check the pod listing on this newly deployed workload cluster.
$ kubectl get pods -A NAMESPACE NAME READY STATUS RESTARTS AGE avi-system ako-0 1/1 Running 0 16h kube-system antrea-agent-wtmvb 2/2 Running 0 16h kube-system antrea-agent-zngvx 2/2 Running 0 16h kube-system antrea-controller-556dc8489c-zl47z 1/1 Running 0 16h kube-system coredns-68d49685bd-46wq6 1/1 Running 0 16h kube-system coredns-68d49685bd-nsrpm 1/1 Running 0 16h kube-system etcd-workload3a-control-plane-28n6c 1/1 Running 0 16h kube-system kube-apiserver-workload3a-control-plane-28n6c 1/1 Running 0 16h kube-system kube-controller-manager-workload3a-control-plane-28n6c 1/1 Running 0 16h kube-system kube-proxy-lfmc7 1/1 Running 0 16h kube-system kube-proxy-tvsbc 1/1 Running 0 16h kube-system kube-scheduler-workload3a-control-plane-28n6c 1/1 Running 0 16h kube-system kube-vip-workload3a-control-plane-28n6c 1/1 Running 0 16h kube-system metrics-server-79476747c9-g6rmf 1/1 Running 0 16h kube-system vsphere-cloud-controller-manager-xsdcj 1/1 Running 0 16h kube-system vsphere-csi-controller-fd4b88d9b-j6rpm 6/6 Running 0 16h kube-system vsphere-csi-node-jbzpl 3/3 Running 0 16h kube-system vsphere-csi-node-kjzpc 3/3 Running 0 16h pinniped-concierge pinniped-concierge-588c757cf4-2qbpv 1/1 Running 0 16h pinniped-concierge pinniped-concierge-588c757cf4-56k4s 1/1 Running 0 16h pinniped-concierge pinniped-concierge-kube-cert-agent-d51497f6 1/1 Running 0 16h pinniped-supervisor pinniped-post-deploy-job-ckxqt 0/1 Error 0 16h pinniped-supervisor pinniped-post-deploy-job-xv2zr 0/1 Completed 0 16h tkg-system kapp-controller-5b4756cfb-84hj2 1/1 Running 0 16h vmware-system-tmc agent-updater-59b85d846d-7zftx 1/1 Running 0 16h vmware-system-tmc agentupdater-workload-1633592640-b68lz 0/1 Completed 0 43s vmware-system-tmc cluster-auth-pinniped-6b99b89b6f-bpff8 1/1 Running 0 16h vmware-system-tmc cluster-auth-pinniped-6b99b89b6f-hxwzx 1/1 Running 0 16h vmware-system-tmc cluster-auth-pinniped-kube-cert-agent-79bbdd7c44-49nqx 1/1 Running 0 16h vmware-system-tmc cluster-health-extension-755c5bf45d-kwccn 1/1 Running 0 16h vmware-system-tmc extension-manager-f78b7776d-6s984 1/1 Running 0 16h vmware-system-tmc extension-updater-5bd46745c7-gb6cc 1/1 Running 0 16h vmware-system-tmc gatekeeper-operator-manager-9bf45f5f5-r7lhf 1/1 Running 0 16h vmware-system-tmc inspection-extension-5c7567d669-7b77t 1/1 Running 0 16h vmware-system-tmc intent-agent-77d75dd59f-jb6xp 1/1 Running 0 16h vmware-system-tmc logs-collector-cluster-auth-pinniped-20211006150814-pngks 0/1 Completed 0 16h vmware-system-tmc logs-collector-cluster-health-extension-20211006150814-k59zt 0/1 Completed 0 16h vmware-system-tmc logs-collector-extension-manager-20211006150815-g2ghh 0/1 Completed 0 16h vmware-system-tmc logs-collector-gatekeeper-operator-20211006150814-wmq5b 0/1 Completed 0 16h vmware-system-tmc logs-collector-inspection-20211006150814-s8t8k 0/1 Completed 0 16h vmware-system-tmc logs-collector-intent-agent-20211006150814-cx4mn 0/1 Completed 0 16h vmware-system-tmc logs-collector-policy-insight-extension-20211006150814-jcdgz 0/1 Completed 0 16h vmware-system-tmc logs-collector-policy-sync-extension-20211006150814-2f9r5 0/1 Completed 0 16h vmware-system-tmc logs-collector-tmc-observer-20211006150814-xhvnh 0/1 Completed 0 16h vmware-system-tmc policy-insight-extension-manager-5858b4dc68-l2jbx 1/1 Running 0 16h vmware-system-tmc policy-sync-extension-cfcd4b7b5-4lh64 1/1 Running 0 16h vmware-system-tmc sync-agent-f968b579-bq244 1/1 Running 0 16h vmware-system-tmc tmc-observer-67cb4c7fc8-7tq9d 1/1 Running 0 16h
The ako-0 pod in the avi-system namespace provides the NSX ALB integration. The pods in the pinniped-concierge namespace are for the LDAP/AD integration. This looks good! Hopefully you can now appreciate how good this functionality would be for a Kubernetes cluster administrator. It certainly makes access policy management a lot easier when trying to manage the roles of multiple users across multiple Kubernetes clusters.
If you are interested in Tanzu Mission Control, why not check out the newly announced Tanzu Mission Control Starter, a free tier of Tanzu Mission Control. This was just announced at VMworld 2021 along with Tanzu Community Edition. Check it out!