Securing application Ingress access on TKG v1.4 with Cert Manager and Contour
In this article, I will walk through the steps involved in securing application Ingress access on TKG v1.4. To achieve this, I will use 2 packages that are available with TKG v1.4, Cert Manager and Contour. We will deploy a sample application kuard – Kubernetes Up and Running demo, and show how we can use these packages to automatically generated certificates to establish trust between our client (browser) and the application (kuard) which will be accessed via an Ingress. For the purposes of this article, I will create my own local Certificate Authority. If you have access to a valid CA, you might prefer to use that instead. At the end of this procedure, we should see how a user can point their browser to the FQDN of the application, and connect to it without the browser reporting any security warnings.
Note that there are some requirements before we begin. The assumption is that you have already deployed a TKG v1.4 workload cluster to work with, along with the Carvel Tools, and have kubectl available to query the cluster. A Load Balancer service is also required. This Tanzu Kubernetes Grid deployment is on vSphere so I am using the NSX Advanced Load Balancer in this example. Note that I am doing all of these operations from an Ubuntu 20.04.2 distro. Some commands, such as openssl, might be subtly different if you are on another distro.
Step 1 – Install Cert Manager
The first step is to install the Cert Manager package onto the workload cluster. No bespoke parameters or changes are needed. We can simply deploy the default.
$ tanzu package available get cert-manager.tanzu.vmware.com - Retrieving package details for cert-manager.tanzu.vmware.com... NAME: cert-manager.tanzu.vmware.com DISPLAY-NAME: cert-manager SHORT-DESCRIPTION: Certificate management PACKAGE-PROVIDER: VMware LONG-DESCRIPTION: Provides certificate management provisioning within the cluster MAINTAINERS: [{Nicholas Seemiller}] SUPPORT: Support provided by VMware for deployment on TKG 1.4+ clusters. Best-effort support for deployment on any conformant Kubernetes cluster. Contact support by opening a support request via VMware Cloud Services or my.vmware.com. CATEGORY: [certificate management] $ tanzu package available list cert-manager.tanzu.vmware.com - Retrieving package versions for cert-manager.tanzu.vmware.com... NAME VERSION RELEASED-AT cert-manager.tanzu.vmware.com 1.1.0+vmware.1-tkg.2 2020-11-24T18:00:00Z $ tanzu package install cert-manager --package-name cert-manager.tanzu.vmware.com --version 1.1.0+vmware.1-tkg.2 / Installing package 'cert-manager.tanzu.vmware.com' | Getting namespace 'default' | Getting package metadata for 'cert-manager.tanzu.vmware.com' | Creating service account 'cert-manager-default-sa' | Creating cluster admin role 'cert-manager-default-cluster-role' | Creating cluster role binding 'cert-manager-default-cluster-rolebinding' - Creating package resource | Package install status: Reconciling Added installed package 'cert-manager' in namespace 'default' $ tanzu package installed list / Retrieving installed packages... NAME PACKAGE-NAME PACKAGE-VERSION STATUS cert-manager cert-manager.tanzu.vmware.com 1.1.0+vmware.1-tkg.2 Reconcile succeeded $ kubectl get all -n cert-manager NAME READY STATUS RESTARTS AGE pod/cert-manager-8576785984-2rtbm 1/1 Running 0 119s pod/cert-manager-cainjector-dbdb9546-52v66 1/1 Running 0 119s pod/cert-manager-webhook-5f688f647b-spxqx 1/1 Running 0 119s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/cert-manager ClusterIP 100.66.243.200 <none> 9402/TCP 119s service/cert-manager-webhook ClusterIP 100.70.82.204 <none> 443/TCP 119s NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/cert-manager 1/1 1 1 119s deployment.apps/cert-manager-cainjector 1/1 1 1 119s deployment.apps/cert-manager-webhook 1/1 1 1 119s NAME DESIRED CURRENT READY AGE replicaset.apps/cert-manager-8576785984 1 1 1 119s replicaset.apps/cert-manager-cainjector-dbdb9546 1 1 1 119s replicaset.apps/cert-manager-webhook-5f688f647b 1 1 1 119s
Step 2 – Install Contour
Contour is an open-source Ingress controller from VMware. We need to provide a small change to its default configuration so that it uses a Load Balancer service (provide by NSX ALB). We do this by passing a values manifest file at deployment time, as shown below.
$ cat contour-simple.yaml envoy: service: type: LoadBalancer certificates: useCertManager: true $ tanzu package available list contour.tanzu.vmware.com - Retrieving package versions for contour.tanzu.vmware.com... NAME VERSION RELEASED-AT contour.tanzu.vmware.com 1.17.1+vmware.1-tkg.1 2021-07-23T18:00:00Z $ tanzu package install contour -p contour.tanzu.vmware.com --version 1.17.1+vmware.1-tkg.1 --values-file contour-simple.yaml / Installing package 'contour.tanzu.vmware.com' | Getting namespace 'default' | Getting package metadata for 'contour.tanzu.vmware.com' | Creating service account 'contour-default-sa' | Creating cluster admin role 'contour-default-cluster-role' | Creating cluster role binding 'contour-default-cluster-rolebinding' | Creating secret 'contour-default-values' - Creating package resource \ Package install status: Reconciling Added installed package 'contour' in namespace 'default' $ tanzu package installed list - Retrieving installed packages... NAME PACKAGE-NAME PACKAGE-VERSION STATUS cert-manager cert-manager.tanzu.vmware.com 1.1.0+vmware.1-tkg.2 Reconcile succeeded contour contour.tanzu.vmware.com 1.17.1+vmware.1-tkg.1 Reconcile succeeded $ kubectl get pods -A | grep 'contour\|envoy' tanzu-system-ingress contour-5dccf8dd5f-dqdbh 1/1 Running 0 93s tanzu-system-ingress contour-5dccf8dd5f-s5pvc 1/1 Running 0 93s tanzu-system-ingress envoy-q9nt8 2/2 Running 0 94s $ kubectl get svc -A | grep envoy tanzu-system-ingress envoy LoadBalancer 100.68.55.154 xx.yy.112.159 80:30298/TCP,443:31028/TCP 104s
Cert Manager and Contour are now successfully deployed, and Envoy (part of Contour) has received a Load Balancer IP address (which I have partially obfuscated). We are good to proceed to the next steps.
Step 3 – Configure DNS
Next, add this VIP/LB IP Address of envoy to your DNS so that the application you plan to create (in my case, Kuard – Kubernetes Up and Running demo) resolves to it. For this demonstration, I am using the FDQN kuard.corinternal.com.
$ nslookup kuard.corinternal.com Server: 127.0.0.53 Address: 127.0.0.53#53 Non-authoritative answer: Name: kuard.corinternal.com Address: xx.yy.112.159
Step 4 – Setup a local Certificate Authority
Not everyone has access to a valid Certificate Authority. Thus, you can follow these steps to create your own local CA. There are 3 steps in total, as you will need to create a non-password-protected key that can be used by Kubernetes. For more details about creating a local CA, check out this very good blog from my colleague Jorge. Note that the default Ubuntu OpenSSL configuration had all of the necessary v3 extensions in place, so I did not need to change anything. This may not be the case on your particular distro, so again, refer to Jorge’s blog on how to do that if necessary.
- Create a private key: openssl genrsa -des3 -out localRootCA.key 2048
- Create the CA with SSLv3 extensions using the private key from the previous step: openssl req -x509 -new -nodes -key localRootCA.key -reqexts v3_req -extensions v3_ca -sha256 -days 1825 -out localRootCA.pem
- The command in step 2 created a password protected CA key. This command creates a password-unprotected version: openssl rsa -in localRootCA.key -out unprotected-localRootCA.key. A password unprotected version is needed for the Kubernetes secret in the next step.
Step 5 – Create a K8s secret with the CA info
In this step, we take the CA files created in step 4 and build a Kubernetes TLS secret from them.
$ kubectl create secret tls my-ca-secret --key unprotected-localRootCA.key --cert localRootCA.pem -n cert-manager secret/my-ca-secret created
Step 6 – Deploy Kuard
We can now roll out the application. Here is a sample manifest to deploy the kuard application, along with what you should observe after it has been deployed. Note that this application has no external ip address associated with it.
$ cat kuard.yaml apiVersion: apps/v1 kind: Deployment metadata: name: kuard spec: selector: matchLabels: app: kuard replicas: 1 template: metadata: labels: app: kuard spec: containers: - image: gcr.io/kuar-demo/kuard-amd64:1 imagePullPolicy: Always name: kuard ports: - containerPort: 8080 --- apiVersion: v1 kind: Service metadata: name: kuard spec: ports: - port: 80 targetPort: 8080 protocol: TCP selector: app: kuard $ kubectl apply -f kuard.yaml deployment.apps/kuard created service/kuard created $ kubectl get deploy,svc NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/kuard 1/1 1 1 27s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/kuard ClusterIP 100.70.248.113 <none> 80/TCP 26s service/kubernetes ClusterIP 100.64.0.1 <none> 443/TCP 24m
Step 7 – Create a Certificate Issuer
Now we want to be able to access this application externally, but we also want to access it securely. We can do this via an Ingress. The first thing we need to do is create a ClusterIssuer for the certificates. Note that in the spec below is a reference to a Certificate Authority (CA) and the secret is the one we created earlier in step 5. We will make this a cluster-wide ClusterIssuer. Otherwise you will need to make an Issuer object on every namespace. Here is the manifest that you need to apply.
$ cat ca-issuer.yaml apiVersion: cert-manager.io/v1 kind: ClusterIssuer metadata: name: ca-issuer namespace: cert-manager spec: ca: secretName: my-ca-secret $ kubectl apply -f ca-issuer.yaml clusterissuer.cert-manager.io/ca-issuer created
Step 8 – Create the Ingress
In this step, we will create the Ingress with certificate annotations that point to our previously created ClusterIssuer. The Ingress class is Contour, and this is identified via another annotation. When there is a request to the kuard application, identified by its FQDN (kuard.corinternal.com), through the Ingress, a certificate will be automatically created by cert manager and stored in the secret myapp-cert. If everything is working correctly, the Ingress for our application should show the Envoy Load Balancer IP address.
$ cat ingress-with-cert.yaml apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: kuard annotations: kubernetes.io/ingress.class: "contour" cert-manager.io/cluster-issuer: "ca-issuer" spec: defaultBackend: service: name: kuard port: number: 80 rules: - host: kuard.corinternal.com http: paths: - path: / pathType: Prefix backend: service: name: kuard port: number: 80 tls: - hosts: - kuard.corinternal.com secretName: myapp-cert $ kubectl apply -f ingress-with-cert.yaml ingress.networking.k8s.io/kuard created $ kubectl get ingress NAME CLASS HOSTS ADDRESS PORTS AGE kuard <none> kuard.corinternal.com xx.yy.112.159 80, 443 4s
Step 9 – Verify everything is working
Now that the Ingress has been successfully created, we can now try to access our application via a browser, such as Firefox. If you tried to access this application through an Ingress that had no certificate integration, you will probably observe a security risk warning and a message about self-signed certificates – something like this:
Now with our own local CA in place for certificates, the message should change to something like this:
The reason for this error is that we created our own local CA, and the browser will not know who issued that certificate. If you used a recognized certificate authority, you should not observe this if the browser has the issuer in its list of CAs. What we can do to avoid this warning is import our CA into to the CA that the browser trusts.
Once the local CA has been added to the browser, I can now connect to my application without any errors.
And now we are successfully using the Cert Manager and Contour Carvel packages available in TKG to establish trust between our client (browser) and the application running in Kubernetes. Hope you find this useful.
The manifests used in this post can be found here on GitHub.
