In this post, I will look at how to secure LDAP communication using TLS certificates. LDAP integration with a TKG cluster is provided via Dex and Pinniped packages. Note that TKG already uses a self-signed Issuer to generate TLS certificates to secure HTTPS traffic to Pinniped and Dex by default; Issuers and ClusterIssuers represent certificate authorities (CAs) that are able to generate signed certificates, and are available via the Cert Manager package. This default Issuer method can be modified. In the official documentation, two alternate methods to secure HTTPS traffic are offered. The first method suggests creating your own TLS secret. This is a little complex as it involves the manual creation of Certificate Signing Requests (CSRs), then the creation of certificates with X509 extensions, before eventually wrapping the certificates up in Kubernetes secrets. A more elegant approach is offered through the use of a ClusterIssuer which can be created with a specific CA. Both methods require a Pinniped reconfiguration, as we shall see. This procedure assumes that you have already deployed a TKG v1.4 management cluster with LDAP already integrated. It also assumes availability of a Load Balancer. This deployment is on vSphere, so the NSX Advanced Load Balancer provides this functionality. I will now step through the second method suggested by the official docs, and see how to create a ClusterIssuer with a custom CA to secure Pinniped and Dex HTTPS communication.
1. Use LoadBalancer Services for Pinniped and DEX
Before we begin integrating LDAP and secure communication with certificates, we need to convert the Pinniped and Dex services from NodePort to LoadBalancer. Thus, an NSX Advanced Load Balancer (or similar) is required when deploying TKG on vSphere. I have outlined these steps in an earlier post. Once the services are of type LoadBalancer, the pinniped post deploy job must be deleted. Wait for the job to successfully restart before proceeding. This can take some time (3-4 minutes). I usually use a watch (-w) to monitor its restart.
$ kubectl get jobs -n pinniped-supervisor NAME COMPLETIONS DURATION AGE pinniped-post-deploy-job 1/1 3m53s 14h $ kubectl delete jobs pinniped-post-deploy-job -n pinniped-supervisor job.batch "pinniped-post-deploy-job" deleted $ kubectl get jobs -n pinniped-supervisor -w NAME COMPLETIONS DURATION AGE pinniped-post-deploy-job 0/1 0s pinniped-post-deploy-job 0/1 0s pinniped-post-deploy-job 0/1 0s 0s pinniped-post-deploy-job 1/1 11s 11s
2. Create a local Certificate Authority (CA)
If you already have a CA, then this step is not necessary. However, since many readers use home labs or test environments which may not have access to a Certificate Service to retrieve a CA, we can simply create our own local CA for the purposes of this exercise.
As mentioned, Issuers and ClusterIssuers are K8s resources that represent certificate authorities (CAs), and are available through Cert Manager (which is already installed on the TKG management cluster). Issuers and ClusterIssuers generate signed certificates when they receive certificate signing requests (CRs). All certificates created in Cert Manager require an issuer. This is what we are creating here, and the resulting ClusterIssuer, which will be referenced by Pinniped and Dex, handles the CSRs and Certificates.
I am creating my own local CA using the openssl command. First create a private key, and use the private key to create the CA. Note that the last command creates an unprotected version of the CA (no pass phrase) which is used to create a K8s secret in the next step.
$ openssl genrsa -des3 -out localRootCA.key 2048 Generating RSA private key, 2048 bit long modulus ................................................................+++ .........................................................................................................+++ e is 65537 (0x10001) Enter pass phrase for localRootCA.key: ********** Verifying - Enter pass phrase for localRootCA.key: ********** $ openssl req -x509 -new -nodes -key localRootCA.key -reqexts v3_req -extensions v3_ca -sha256 -days 1825 -out localRootCA.pem Enter pass phrase for localRootCA.key: *********** You are about to be asked to enter information that will be incorporated into your certificate request. What you are about to enter is what is called a Distinguished Name or a DN. There are quite a few fields but you can leave some blank For some fields there will be a default value, If you enter '.', the field will be left blank. ----- Country Name (2 letter code) [AU]:IE State or Province Name (full name) [Some-State]:Cork Locality Name (eg, city) []:Cork Organization Name (eg, company) [Internet Widgits Pty Ltd]:VMware Organizational Unit Name (eg, section) []:OCTO Common Name (e.g. server FQDN or YOUR name) []:pks-cli.rainpole.com Email Address []:chogan@rainpole.com $ openssl rsa -in localRootCA.key -out unprotected-localRootCA.key Enter pass phrase for localRootCA.key: *********** writing RSA key
3. Create a K8s secret that contains the local CA
This secret contains the CA that does not have a pass phrase, created in the last command of the previous step. Note that Cert Manager is already deployed as part of the TKG management cluster. The secret is created in the Cert Manager namespace.
$ kubectl create secret tls my-ca-secret --key unprotected-localRootCA.key --cert localRootCA.pem \ -n cert-manager secret/my-ca-secret created
4. Create the ClusterIssuer
We can now create the ClusterIssuer, using the secret containing the CA from the previous step.
$ cat ca-issuer.yaml apiVersion: cert-manager.io/v1 kind: ClusterIssuer metadata: name: ca-issuer namespace: cert-manager spec: ca: secretName: my-ca-secret $ kubectl apply -f ca-issuer.yaml clusterissuer.cert-manager.io/ca-issuer created $ kubectl get ClusterIssuer NAME READY AGE ca-issuer True 9s $ kubectl describe ClusterIssuer Name: ca-issuer Namespace: Labels: <none> Annotations: <none> API Version: cert-manager.io/v1 Kind: ClusterIssuer Metadata: Creation Timestamp: 2021-11-22T13:55:38Z Generation: 1 Managed Fields: API Version: cert-manager.io/v1 Fields Type: FieldsV1 fieldsV1: f:status: .: f:conditions: Manager: controller Operation: Update Time: 2021-11-22T13:55:38Z API Version: cert-manager.io/v1 Fields Type: FieldsV1 fieldsV1: f:metadata: f:annotations: .: f:kubectl.kubernetes.io/last-applied-configuration: f:spec: .: f:ca: .: f:secretName: Manager: kubectl-client-side-apply Operation: Update Time: 2021-11-22T13:55:38Z Resource Version: 10192 UID: a1134f01-769d-4648-a869-80ddca520ef9 Spec: Ca: Secret Name: my-ca-secret Status: Conditions: Last Transition Time: 2021-11-22T13:55:38Z Message: Signing CA verified Reason: KeyPairVerified Status: True Type: Ready Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal KeyPairVerified <invalid> (x2 over <invalid>) cert-manager Signing CA verified
5. Modify Pinniped secret to use ClusterIssuer
We now need to retrieve the Pinniped add-on secret, retrieve the encoded values.yaml string, convert it from base64 to a human readable format, add the option to use a ClusterIssuer. The field custom_cluster_issuer in the values.yaml must be set to our ClusterIssuer. Then we need to re-encode the values.yaml string, add it to the Pinniped add-on secret and apply the new secret.
5.1 Retrieve the Pinniped add-on secret
Note that the name of the secret will be different to the command shown below as it contains the name of your management cluster. I am redirecting the secret (in yaml format) to a file so that I can read and modify it. To make it easier to read, many of the encoded strings below have been truncated, as has the contents of the file. The values.yaml string is highlighted in blue below. This is what needs updating.
$ kubectl get secret ldap-cert-pinniped-addon -n tkg-system -o yaml > ldap-cert-pinniped-addon-secret.yaml $ more ldap-cert-pinniped-addon-secret.yaml apiVersion: v1 data: overlays.yaml: I0Agb...wcwo= values.yaml: I0BkYX...xlLmNvbQo= kind: Secret metadata: annotations: kubectl.kubernetes.io/last-applied-configuration: | ...
5.2 Decode the values.yaml base64 string
Decode the values.yaml string since it is encoded in base64. I normally copy and paste it into a file to make it easier to work with. You can examine the configuration by using the base64 -d command on Linux. Currently, there should be no “custom_cluster_issuer” value.
$ vi pinniped-config-values.base64 # paste the contents of the values.yaml string in here $ cat pinniped-config-values.base64 | base64 -d #@data/values #@overlay/match-child-defaults missing_ok=True --- infrastructure_provider: vsphere tkg_cluster_role: management custom_cluster_issuer: "" custom_tls_secret: "" http_proxy: "" https_proxy: "" no_proxy: "" identity_management_type: ldap ...
5.3 Save the decoded values.yaml to a manifest for editing
Store the decoded contents into another file so that it can be edited.
$ cat pinniped-config-values.base64 | base64 -d > pinniped-config-values.yaml
5.4 Add the custom_cluster_issuer to the Pinniped configuration
The field custom_cluster_issuer in the values.yaml must be set to the name of ClusterIssuer created in an earlier step. In this procedure, I called it ca-issuer.
#@data/values #@overlay/match-child-defaults missing_ok=True --- infrastructure_provider: vsphere tkg_cluster_role: management custom_cluster_issuer: "ca-issuer" custom_tls_secret: "" http_proxy: "" https_proxy: "" no_proxy: "" identity_management_type: ldap ...
5.5 Re-encode the updated Pinniped configuration
We now need to convert the values.yaml configuration back into a base64 encoded string. The following command will store it in a file but you could just display it directly in the shell.
$ cat pinniped-config-values.yaml | base64 -w 0 > encoded.base64 $ cat encoded.base64 I0BkYXRhL3ZhbHVlcwoj...NvbQo=
5.6 Add the encoded configuration back to the Pinniped add-on secret
Copy the contents of the base64 encoded. values.yaml string. The addon-secret manifest that we captured a number of steps ago now needs to be updated with a new values.yaml string that has the new configuration (it now contains the custom_tls_secret setting). Replace the current values.yaml string with the new string from the previous step.
$ vi ldap-cert-pinniped-addon-secret.yaml apiVersion: v1 data: overlays.yaml: I0Agb...wcwo= values.yaml: I0BkYX...xlLmNvbQo= # paste the new encoded values.yaml string in here kind: Secret metadata: annotations: kubectl.kubernetes.io/last-applied-configuration: | ...
5.7 Apply the updated Pinniped configuration / add-on secret
Change the add-on secret in the cluster. Simply apply our updated add-on secret with its new values.yaml.
$ kubectl apply -f ldap-cert-pinniped-addon-secret.yaml secret/ldap-cert-pinniped-addon configured
5.8 Monitor Pinniped Reconcile
Ensure that there are no issues with the new configuration by monitoring the Pinniped application. It needs to go through a reconcile to implement the configuration change. Make sure the reconcile succeeds. You should see it go through a number of Reconciling states before eventually reconciling.
$ kubectl get app pinniped -n tkg-system -w NAME DESCRIPTION SINCE-DEPLOY AGE pinniped Reconcile succeeded 3m20s 15h pinniped Reconciling 16s 15h pinniped Reconciling 18s 15h pinniped Reconciling 20s 15h pinniped Reconciling 22s 15h pinniped Reconciling 24s 15h pinniped Reconciling 26s 15h pinniped Reconciling 28s 15h pinniped Reconciling 30s 15h pinniped Reconciling 32s 15h pinniped Reconciling 34s 15h pinniped Reconciling 36s 15h pinniped Reconciling 38s 15h pinniped Reconciling 40s 15h pinniped Reconciling 42s 15h pinniped Reconciling 44s 15h pinniped Reconciling 44s 15h pinniped Reconcile succeeded 44s 15h
6. Recreate non-admin context / access cluster as LDAP user
Pinniped and Dex are now configured to use the ClusterIssuer for Certificate Signing Requests (CSRs), and they should be sent signed Certificates from Cert Manager using the CA that was provided to the ClusterIssuer. To check that it is working, we should create (or recreate) the non-admin TKG management context.
If you are on an SSH session or non-graphical node, you will probably experience something similar to the what is shown below, where there is “no DISPLAY environment variable specified“. This is because the browser required for Dex authentication cannot be launched in this environment. This is ok, and is expected. I covered this in another post, but will add the steps here once more. We simply remove the non-admin context, and set the TANZU_CLI_PINNIPED_AUTH_LOGIN_SKIP_BROWSER environment variable, and then recreate the non-admin context. This allows us to do a Dex authentication on a different host which has a browser other than the host we are managing the TKG cluster cluster from.
$ kubectl get nodes Error: no DISPLAY environment variable specified ^C $ kubectl config delete-context tanzu-cli-ldap-issuer@ldap-issuer warning: this removed your active context, use "kubectl config use-context" to select a different one deleted context tanzu-cli-ldap-issuer@ldap-issuer from /home/cormac/.kube/config $ export TANZU_CLI_PINNIPED_AUTH_LOGIN_SKIP_BROWSER=true $ tanzu management-cluster kubeconfig get You can now access the cluster by running 'kubectl config use-context tanzu-cli-ldap-issuer@ldap-issuer' $ kubectl config use-context tanzu-cli-ldap-issuer@ldap-issuer Switched to context "tanzu-cli-ldap-issuer@ldap-issuer". $ kubectl get nodes Please log in: https://xx.xx.xx.22/oauth2/authorize?access_type=offline&client_id=pinniped-cli\ &code_challenge=NrihaWN8AmYcBiGGLrzPlPLAbN-Rp5pBibztRtucivw&code_challenge_method=S256&nonce=\ 347caf071934368b6f758dbb47f62fb5&redirect_uri=http%3A%2F%2F127.0.0.1%3A44531%2Fcallback&\ response_type=code&scope=offline_access+openid+pinniped%3Arequest-audience&state=\ 8ccc783a86bfba1e1701022b4e50d3bb
Copy and paste the above “Please log in” link into a browser on a host which can reach the Pinniped and Dex services. This will present you with the standard Dex auth window where you can add the LDAP user that you wish to give access to the TKG cluster to. After adding the credentials, the browser will redirect to the IP address of the Dex service, but then it attempts to do a call back to the localhost. This will fail, and it will report an error about unable to establish a localhost connection. This is expected, as it is trying to reach the localhost where the kubectl command was run. The final step is to take the callback URL, and on the host where the kubectl command was run and the “Please log in” prompt was observed, open another shell session and run the equivalent curl command using the local host call back URL:
$ curl -L 'http://127.0.0.1:35259/callback?code=4bExu5_NESSuaY7kSCCIGcn8arVwJiVUC19UTZL-Jck.\
61FRn2BeHo7C8fYGbA0aranDsAFT3v0bRTTwq-TizA8&scope=openid+offline_access+pinniped%3Arequest-\
audience&state=edd3196e00a3eb01403d2d4e2d918fa3'
After running the command, a “you have been logged in and may now close this tab” message should appear where the curl command was run. Unless you have already created a ClusterRoleBinding for the LDAP user you authenticated, the kubectl get nodes command run previously should error out with the following message:
Error from server (Forbidden): nodes is forbidden: User "chogan@rainpole.com" cannot list \
resource "nodes" in API group "" at the cluster scope
The user chogan@rainpole.com is the LDAP user that I authenticated in the Dex browser. Again, this error is expected as this LDAP user has not been given any privileges on the TKG management cluster.
7. Create a ClusterRoleBinding for the LDAP user
$ kubectl config use-context ldap-cert-admin@ldap-cert Switched to context "ldap-cert-admin@ldap-cert". $ kubectl config get-contexts CURRENT NAME CLUSTER AUTHINFO NAMESPACE * ldap-cert-admin@ldap-cert ldap-cert ldap-cert-admin tanzu-cli-ldap-cert@ldap-cert ldap-cert tanzu-cli-ldap-cert $ cat chogan-crb.yaml kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: chogan subjects: - kind: User name: chogan@rainpole.com apiGroup: roleRef: kind: ClusterRole name: cluster-admin apiGroup: rbac.authorization.k8s.io $ kubectl apply -f chogan-crb.yaml clusterrolebinding.rbac.authorization.k8s.io/chogan created $ kubectl get clusterrolebinding chogan NAME ROLE AGE chogan ClusterRole/cluster-admin 6m $ kubectl config use-context tanzu-cli-ldap-cert@ldap-cert Switched to context "tanzu-cli-ldap-cert@ldap-cert". $ kubectl get nodes NAME STATUS ROLES AGE VERSION ldap-cert-control-plane-77g8v Ready control-plane,master 86m v1.21.2+vmware.1 ldap-cert-md-0-b7f799d64-kgcqm Ready <none> 85m v1.21.2+vmware.1
The authenticated LDAP user (chogan@rainpole.com) is now able to manage the cluster using the non-admin context. The CSRs and Certificates generated for Pinniped and Dex can be examined via the commands below. Note that there are 2 certificate requests related to CAs. One is the original Issuer (e.g. pinniped-selfsigned-ca-issuer) but the other should be the result of our customization; a ClusterIssuer (e.g. ca-issuer). You can describe the CA certificate requests to make certain. Similarly, if you examine the certificates using the describe option, you should see that the certificate is issued by the ClusterIssuer. This will verify that the ClusterIssuer is working as expected.
$ kubectl get certificaterequest -n pinniped-supervisor NAME READY AGE pinniped-ca-4mdtl True 53m pinniped-ca-6nw4z True 78m pinniped-cert-67w7c True 65m pinniped-cert-c24l6 True 78m pinniped-cert-rnckf True 76m pinniped-cert-zp9bj True 53m $ kubectl get certificates -n pinniped-supervisor NAME READY SECRET AGE pinniped-ca True pinniped-ca-key-pair 78m pinniped-cert True pinniped-supervisor-default-tls-certificate 78m $ kubectl get certificaterequest -n tanzu-system-auth NAME READY AGE dex-ca-ffv4q True 79m dex-ca-rm8mf True 53m dex-cert-j22n2 True 65m dex-cert-kfvgh True 76m dex-cert-ktntv True 53m dex-cert-zlxlv True 79m $ kubectl get certificates -n tanzu-system-auth NAME READY SECRET AGE dex-ca True dex-ca-key-pair 79m dex-cert True dex-cert-tls 79m