Over the last month or so, I have looked at various ways of securing Tanzu Kubernetes Grid (TKG) clusters. One recent post covered the integration of LDAP through Dex and Pinniped so you can control who can access the the non-admin context of your TKG cluster. I’ve also looked at how TKG clusters that do not have direct access to the internet can use a HTTP/HTTPS proxy. Similarly,  I looked at some tips when deploying TKG in an air-gapped environment, pulling all the necessary images from our external image registry and pushing them to a local Harbor registry. In another post, I looked at how TLS certificates can be used implement secure ingress connectivity to an application running in a TKG cluster. And finally I examined network policies that are available via the Antrea CNI (Container Network Interface). These policies can be managed from within a TKG cluster or from Tanzu Mission Control (TMC).As you can see, there are lots of options to secure (or harden) a TKG cluster.

In this post, I will look at how to secure LDAP communication using TLS certificates. LDAP integration with a TKG cluster is provided via Dex and Pinniped packages. Note that TKG already uses a self-signed Issuer to generate TLS certificates to secure HTTPS traffic to Pinniped and Dex by default; Issuers and ClusterIssuers represent certificate authorities (CAs) that are able to generate signed certificates, and are available via the Cert Manager package. This default Issuer method can be modified. In the official documentation, two alternate methods to secure HTTPS traffic are offered. The first method suggests creating your own TLS secret. This is a little complex as it involves the manual creation of Certificate Signing Requests (CSRs), then the creation of certificates with X509 extensions, before eventually wrapping the certificates up in Kubernetes secrets. A more elegant approach is offered through the use of a ClusterIssuer which can be created with a specific CA.  Both methods require a Pinniped reconfiguration, as we shall see. This procedure assumes that you have already deployed a TKG v1.4 management cluster with LDAP already integrated. It also assumes availability of a Load Balancer.  This deployment is on vSphere, so the NSX Advanced Load Balancer provides this functionality. I will now step through the second method suggested by the official docs, and see how to create a ClusterIssuer with a custom CA to secure Pinniped and Dex HTTPS communication.

1. Use LoadBalancer Services for Pinniped and DEX

Before we begin integrating LDAP and secure communication with certificates, we need to convert the Pinniped and Dex services from NodePort to LoadBalancer. Thus, an NSX Advanced Load Balancer (or similar) is required when deploying TKG on vSphere. I have outlined these steps in an earlier post. Once the services are of type LoadBalancer, the pinniped post deploy job must be deleted.  Wait for the job to successfully restart before proceeding. This can take some time (3-4 minutes). I usually use a watch (-w) to monitor its restart.

$ kubectl get jobs -n pinniped-supervisor
NAME                       COMPLETIONS   DURATION   AGE
pinniped-post-deploy-job   1/1           3m53s      14h


$ kubectl delete jobs pinniped-post-deploy-job -n pinniped-supervisor
job.batch "pinniped-post-deploy-job" deleted


$ kubectl get jobs -n pinniped-supervisor -w
NAME                       COMPLETIONS   DURATION   AGE
pinniped-post-deploy-job   0/1                      0s
pinniped-post-deploy-job   0/1                      0s
pinniped-post-deploy-job   0/1           0s         0s
pinniped-post-deploy-job   1/1           11s        11s

2. Create a local Certificate Authority (CA)

If you already have a CA, then this step is not necessary. However, since many readers use home labs or test environments which may not have access to a Certificate Service to retrieve a CA, we can simply create our own local CA for the purposes of this exercise.

As mentioned, Issuers and ClusterIssuers are K8s resources that represent certificate authorities (CAs), and are available through Cert Manager (which is already installed on the TKG management cluster). Issuers and ClusterIssuers generate signed certificates when they receive certificate signing requests (CRs). All certificates created in Cert Manager require an issuer. This is what we are creating here, and the resulting ClusterIssuer, which will be referenced by Pinniped and Dex, handles the CSRs and Certificates.

I am creating my own local CA using the openssl command. First create a private key, and use the private key to create the CA. Note that the last command creates an unprotected version of the CA (no pass phrase) which is used to create a K8s secret in the next step.

$ openssl genrsa -des3 -out localRootCA.key 2048
Generating RSA private key, 2048 bit long modulus
................................................................+++
.........................................................................................................+++
e is 65537 (0x10001)
Enter pass phrase for localRootCA.key: **********
Verifying - Enter pass phrase for localRootCA.key: **********


$ openssl req -x509 -new -nodes -key localRootCA.key -reqexts v3_req -extensions v3_ca -sha256 -days 1825 -out localRootCA.pem
Enter pass phrase for localRootCA.key: ***********
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [AU]:IE
State or Province Name (full name) [Some-State]:Cork
Locality Name (eg, city) []:Cork
Organization Name (eg, company) [Internet Widgits Pty Ltd]:VMware
Organizational Unit Name (eg, section) []:OCTO
Common Name (e.g. server FQDN or YOUR name) []:pks-cli.rainpole.com
Email Address []:chogan@rainpole.com


$ openssl rsa -in localRootCA.key -out unprotected-localRootCA.key
Enter pass phrase for localRootCA.key: ***********
writing RSA key

3. Create a K8s secret that contains the local CA

This secret contains the CA that does not have a pass phrase, created in the last command of the previous step. Note that Cert Manager is already deployed as part of the TKG management cluster. The secret is created in the Cert Manager namespace.

$ kubectl create secret tls my-ca-secret --key unprotected-localRootCA.key --cert localRootCA.pem \
-n cert-manager
secret/my-ca-secret created

4. Create the ClusterIssuer

We can now create the ClusterIssuer, using the secret containing the CA from the previous step.

$ cat ca-issuer.yaml
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: ca-issuer
  namespace: cert-manager
spec:
  ca:
    secretName: my-ca-secret


$ kubectl apply -f ca-issuer.yaml
clusterissuer.cert-manager.io/ca-issuer created


$ kubectl get ClusterIssuer
NAME        READY  AGE
ca-issuer  True    9s


$ kubectl describe ClusterIssuer
Name:        ca-issuer
Namespace:
Labels:      <none>
Annotations:  <none>
API Version:  cert-manager.io/v1
Kind:        ClusterIssuer
Metadata:
  Creation Timestamp:  2021-11-22T13:55:38Z
  Generation:          1
  Managed Fields:
    API Version:  cert-manager.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:status:
        .:
        f:conditions:
    Manager:      controller
    Operation:    Update
    Time:        2021-11-22T13:55:38Z
    API Version:  cert-manager.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .:
          f:kubectl.kubernetes.io/last-applied-configuration:
      f:spec:
        .:
        f:ca:
          .:
          f:secretName:
    Manager:        kubectl-client-side-apply
    Operation:      Update
    Time:            2021-11-22T13:55:38Z
  Resource Version:  10192
  UID:              a1134f01-769d-4648-a869-80ddca520ef9
Spec:
  Ca:
    Secret Name:  my-ca-secret
Status:
  Conditions:
    Last Transition Time:  2021-11-22T13:55:38Z
    Message:               Signing CA verified
    Reason:                KeyPairVerified
    Status:                True
    Type:                  Ready
Events:
  Type    Reason          Age                             From          Message
  ----    ------          ----                            ----          -------
  Normal  KeyPairVerified  <invalid> (x2 over <invalid>)  cert-manager  Signing CA verified

5. Modify Pinniped secret to use ClusterIssuer

We now need to retrieve the Pinniped add-on secret, retrieve the encoded values.yaml string, convert it from base64 to a human readable format, add the option to use a ClusterIssuer. The field custom_cluster_issuer in the values.yaml must be set to our ClusterIssuer. Then we need to re-encode the values.yaml string, add it to the Pinniped add-on secret and apply the new secret.

5.1 Retrieve the Pinniped add-on secret

Note that the name of the secret will be different to the command shown below as it contains the name of your management cluster. I am redirecting the secret (in yaml format) to a file so that I can read and modify it. To make it easier to read, many of the encoded strings below have been truncated, as has the contents of the file. The values.yaml string is highlighted in blue below. This is what needs updating.

$ kubectl get secret ldap-cert-pinniped-addon -n tkg-system -o yaml > ldap-cert-pinniped-addon-secret.yaml

$ more ldap-cert-pinniped-addon-secret.yaml
apiVersion: v1
data:
  overlays.yaml: I0Agb...wcwo=
  values.yaml: I0BkYX...xlLmNvbQo=
kind: Secret
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
...

5.2 Decode the values.yaml base64 string

Decode the values.yaml string since it is encoded in base64. I normally copy and paste it into a file to make it easier to work with. You can examine the configuration by using the base64 -d command on Linux. Currently, there should be no “custom_cluster_issuer” value.

$ vi pinniped-config-values.base64       # paste the contents of the values.yaml string in here


$ cat pinniped-config-values.base64 | base64 -d
#@data/values
#@overlay/match-child-defaults missing_ok=True
---
infrastructure_provider: vsphere
tkg_cluster_role: management
custom_cluster_issuer: ""
custom_tls_secret: ""
http_proxy: ""
https_proxy: ""
no_proxy: ""
identity_management_type: ldap
...

5.3 Save the decoded values.yaml to a manifest for editing

Store the decoded contents into another file so that it can be edited.

$ cat pinniped-config-values.base64 | base64 -d > pinniped-config-values.yaml

5.4 Add the custom_cluster_issuer to the Pinniped configuration

The field custom_cluster_issuer in the values.yaml must be set to the name of ClusterIssuer created in an earlier step. In this procedure, I called it ca-issuer.

#@data/values
#@overlay/match-child-defaults missing_ok=True
---
infrastructure_provider: vsphere
tkg_cluster_role: management
custom_cluster_issuer: "ca-issuer"
custom_tls_secret: ""
http_proxy: ""
https_proxy: ""
no_proxy: ""
identity_management_type: ldap
...

5.5 Re-encode the updated Pinniped configuration

We now need to convert the values.yaml configuration back into a base64 encoded string. The following command will store it in a file but you could just display it directly in the shell.

$ cat pinniped-config-values.yaml | base64 -w 0 > encoded.base64


$ cat encoded.base64
I0BkYXRhL3ZhbHVlcwoj...NvbQo=

5.6 Add the encoded configuration back to the Pinniped add-on secret

Copy the contents of the base64 encoded. values.yaml string. The addon-secret manifest that we captured a number of steps ago now needs to be updated with a new values.yaml string that has the new configuration (it now contains the custom_tls_secret setting). Replace the current values.yaml string with the new string from the previous step.

$ vi ldap-cert-pinniped-addon-secret.yaml     

apiVersion: v1
data:
  overlays.yaml: I0Agb...wcwo=
  values.yaml: I0BkYX...xlLmNvbQo=          # paste the new encoded values.yaml string in here
kind: Secret
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
...

5.7 Apply the updated Pinniped configuration / add-on secret

Change the add-on secret in the cluster. Simply apply our updated add-on secret with its new values.yaml.

$ kubectl apply -f ldap-cert-pinniped-addon-secret.yaml
secret/ldap-cert-pinniped-addon configured

5.8 Monitor Pinniped Reconcile

Ensure that there are no issues with the new configuration by monitoring the Pinniped application. It needs to go through a reconcile to implement the configuration change. Make sure the reconcile succeeds. You should see it go through a number of Reconciling states before eventually reconciling.

$ kubectl get app pinniped -n tkg-system -w
NAME       DESCRIPTION           SINCE-DEPLOY   AGE
pinniped   Reconcile succeeded   3m20s          15h
pinniped   Reconciling           16s            15h
pinniped   Reconciling           18s            15h
pinniped   Reconciling           20s            15h
pinniped   Reconciling           22s            15h
pinniped   Reconciling           24s            15h
pinniped   Reconciling           26s            15h
pinniped   Reconciling           28s            15h
pinniped   Reconciling           30s            15h
pinniped   Reconciling           32s            15h
pinniped   Reconciling           34s            15h
pinniped   Reconciling           36s            15h
pinniped   Reconciling           38s            15h
pinniped   Reconciling           40s            15h
pinniped   Reconciling           42s            15h
pinniped   Reconciling           44s            15h
pinniped   Reconciling           44s            15h
pinniped   Reconcile succeeded   44s            15h

6. Recreate non-admin context / access cluster as LDAP user

Pinniped and Dex are now configured to use the ClusterIssuer for Certificate Signing Requests (CSRs), and they should be sent signed Certificates from Cert Manager using the CA that was provided to the ClusterIssuer. To check that it is working, we should create (or recreate) the non-admin TKG management context.

If you are on an SSH session or non-graphical node, you will probably experience something similar to the what is shown below, where there is “no DISPLAY environment variable specified“. This is because the browser required for Dex authentication cannot be launched in this environment. This is ok, and is expected. I covered this in another post, but will add the steps here once more. We simply remove the non-admin context, and set the TANZU_CLI_PINNIPED_AUTH_LOGIN_SKIP_BROWSER environment variable, and then recreate the non-admin context. This allows us to do a Dex authentication on a different host which has a browser other than the host we are managing the TKG cluster cluster from.

$ kubectl get nodes
Error: no DISPLAY environment variable specified
^C


$ kubectl config delete-context tanzu-cli-ldap-issuer@ldap-issuer
warning: this removed your active context, use "kubectl config use-context" to select a different one
deleted context tanzu-cli-ldap-issuer@ldap-issuer from /home/cormac/.kube/config


$ export TANZU_CLI_PINNIPED_AUTH_LOGIN_SKIP_BROWSER=true


$ tanzu management-cluster kubeconfig get
You can now access the cluster by running 'kubectl config use-context tanzu-cli-ldap-issuer@ldap-issuer'

 
$ kubectl config use-context tanzu-cli-ldap-issuer@ldap-issuer
Switched to context "tanzu-cli-ldap-issuer@ldap-issuer".


$ kubectl get nodes
Please log in: https://xx.xx.xx.22/oauth2/authorize?access_type=offline&client_id=pinniped-cli\
&code_challenge=NrihaWN8AmYcBiGGLrzPlPLAbN-Rp5pBibztRtucivw&code_challenge_method=S256&nonce=\
347caf071934368b6f758dbb47f62fb5&redirect_uri=http%3A%2F%2F127.0.0.1%3A44531%2Fcallback&\
response_type=code&scope=offline_access+openid+pinniped%3Arequest-audience&state=\
8ccc783a86bfba1e1701022b4e50d3bb

Copy and paste the above “Please log in” link into a browser on a host which can reach the Pinniped and Dex services. This will present you with the standard Dex auth window where you can add the LDAP user that you wish to give access to the TKG cluster to. After adding the credentials, the browser will redirect to the IP address of the Dex service, but then it attempts to do a call back to the localhost. This will fail, and it will report an error about unable to establish a localhost connection. This is expected, as it is trying to reach the localhost where the kubectl command was run. The final step is to take the callback URL, and on the host where the kubectl command was run and the “Please log in” prompt was observed, open another shell session and run the equivalent curl command using the local host call back URL:

$ curl -L 'http://127.0.0.1:35259/callback?code=4bExu5_NESSuaY7kSCCIGcn8arVwJiVUC19UTZL-Jck.\
61FRn2BeHo7C8fYGbA0aranDsAFT3v0bRTTwq-TizA8&scope=openid+offline_access+pinniped%3Arequest-\
audience&state=edd3196e00a3eb01403d2d4e2d918fa3'

After running the command,  a “you have been logged in and may now close this tab” message should appear where the curl command was run. Unless you have already created a ClusterRoleBinding for the LDAP user you authenticated, the kubectl get nodes command run previously should error out with the following message:

Error from server (Forbidden): nodes is forbidden: User "chogan@rainpole.com" cannot list \
resource "nodes" in API group "" at the cluster scope

The user chogan@rainpole.com is the LDAP user that I authenticated in the Dex browser. Again, this error is expected as this LDAP user has not been given any privileges on the TKG management cluster.

7. Create a ClusterRoleBinding for the LDAP user

$ kubectl config use-context ldap-cert-admin@ldap-cert
Switched to context "ldap-cert-admin@ldap-cert".


$ kubectl config get-contexts
CURRENT   NAME                            CLUSTER      AUTHINFO                                   NAMESPACE
*         ldap-cert-admin@ldap-cert       ldap-cert    ldap-cert-admin
          tanzu-cli-ldap-cert@ldap-cert   ldap-cert    tanzu-cli-ldap-cert


$ cat chogan-crb.yaml
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: chogan
subjects:
  - kind: User
    name: chogan@rainpole.com
    apiGroup:
roleRef:
  kind: ClusterRole
  name: cluster-admin
  apiGroup: rbac.authorization.k8s.io


$ kubectl apply -f chogan-crb.yaml
clusterrolebinding.rbac.authorization.k8s.io/chogan created


$ kubectl get clusterrolebinding chogan
NAME     ROLE                        AGE
chogan   ClusterRole/cluster-admin   6m


$ kubectl config use-context tanzu-cli-ldap-cert@ldap-cert
Switched to context "tanzu-cli-ldap-cert@ldap-cert".


$ kubectl get nodes
NAME                             STATUS   ROLES                  AGE   VERSION
ldap-cert-control-plane-77g8v    Ready    control-plane,master   86m   v1.21.2+vmware.1
ldap-cert-md-0-b7f799d64-kgcqm   Ready    <none>                 85m   v1.21.2+vmware.1

$ kubectl get certificaterequest -n pinniped-supervisor
NAME                  READY   AGE
pinniped-ca-4mdtl     True    53m
pinniped-ca-6nw4z     True    78m
pinniped-cert-67w7c   True    65m
pinniped-cert-c24l6   True    78m
pinniped-cert-rnckf   True    76m
pinniped-cert-zp9bj   True    53m


$ kubectl get certificates -n pinniped-supervisor
NAME            READY   SECRET                                        AGE
pinniped-ca     True    pinniped-ca-key-pair                          78m
pinniped-cert   True    pinniped-supervisor-default-tls-certificate   78m


$ kubectl get certificaterequest -n tanzu-system-auth
NAME             READY   AGE
dex-ca-ffv4q     True    79m
dex-ca-rm8mf     True    53m
dex-cert-j22n2   True    65m
dex-cert-kfvgh   True    76m
dex-cert-ktntv   True    53m
dex-cert-zlxlv   True    79m


$ kubectl get certificates -n tanzu-system-auth
NAME       READY   SECRET            AGE
dex-ca     True    dex-ca-key-pair   79m
dex-cert   True    dex-cert-tls      79m