Securing application Ingress access on TKG v1.4 with Cert Manager and Contour

In this article, I will walk through the steps involved in securing application Ingress access on TKG v1.4. To achieve this, I will use 2 packages that are available with TKG v1.4, Cert Manager and Contour. We will deploy a sample application kuard – Kubernetes Up and Running demo, and show how we can use these packages to automatically generated certificates to establish trust between our client (browser) and the application (kuard) which will be accessed via an Ingress. For the purposes of this article, I will create my own local Certificate Authority. If you have access to a valid CA, you might prefer to use that instead. At the end of this procedure, we should see how a user can point their browser to the FQDN of the application, and connect to it without the browser reporting any security warnings.

Note that there are some requirements before we begin. The assumption is that you have already deployed a TKG v1.4 workload cluster to work with, along with the Carvel Tools, and have kubectl available to query the cluster. A Load Balancer service is also required. This Tanzu Kubernetes Grid deployment is on vSphere so I am using the NSX Advanced Load Balancer in this example. Note that I am doing all of these operations from an Ubuntu 20.04.2 distro. Some commands, such as openssl, might be subtly different if you are on another distro.

Step 1 – Install Cert Manager

The first step is to install the Cert Manager package onto the workload cluster. No bespoke parameters or changes are needed. We can simply deploy the default.

$ tanzu package available get
- Retrieving package details for
DISPLAY-NAME:         cert-manager
SHORT-DESCRIPTION:    Certificate management
LONG-DESCRIPTION:     Provides certificate management provisioning within the cluster
MAINTAINERS:          [{Nicholas Seemiller}]
SUPPORT:              Support provided by VMware for deployment on TKG 1.4+ clusters. Best-effort support for deployment on any conformant Kubernetes cluster. Contact support by opening a support request via VMware Cloud Services or
CATEGORY:             [certificate management]

$ tanzu package available list
- Retrieving package versions for
  NAME                           VERSION               RELEASED-AT  1.1.0+vmware.1-tkg.2  2020-11-24T18:00:00Z

$ tanzu package install cert-manager --package-name --version 1.1.0+vmware.1-tkg.2
/ Installing package ''
| Getting namespace 'default'
| Getting package metadata for ''
| Creating service account 'cert-manager-default-sa'
| Creating cluster admin role 'cert-manager-default-cluster-role'
| Creating cluster role binding 'cert-manager-default-cluster-rolebinding'
- Creating package resource
| Package install status: Reconciling

 Added installed package 'cert-manager' in namespace 'default'

$ tanzu package installed list
/ Retrieving installed packages...
  NAME          PACKAGE-NAME                   PACKAGE-VERSION       STATUS
  cert-manager  1.1.0+vmware.1-tkg.2  Reconcile succeeded

$ kubectl get all -n cert-manager
NAME                                         READY   STATUS    RESTARTS   AGE
pod/cert-manager-8576785984-2rtbm            1/1     Running   0          119s
pod/cert-manager-cainjector-dbdb9546-52v66   1/1     Running   0          119s
pod/cert-manager-webhook-5f688f647b-spxqx    1/1     Running   0          119s

NAME                           TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
service/cert-manager           ClusterIP   <none>        9402/TCP   119s
service/cert-manager-webhook   ClusterIP    <none>        443/TCP    119s

NAME                                      READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/cert-manager              1/1     1            1           119s
deployment.apps/cert-manager-cainjector   1/1     1            1           119s
deployment.apps/cert-manager-webhook      1/1     1            1           119s

NAME                                               DESIRED   CURRENT   READY   AGE
replicaset.apps/cert-manager-8576785984            1         1         1       119s
replicaset.apps/cert-manager-cainjector-dbdb9546   1         1         1       119s
replicaset.apps/cert-manager-webhook-5f688f647b    1         1         1       119s

Step 2 – Install Contour

Contour is an open-source Ingress controller from VMware. We need to provide a small change to its default configuration so that it uses a Load Balancer service (provide by NSX ALB). We do this by passing a values manifest file at deployment time, as shown below.

$ cat contour-simple.yaml
    type: LoadBalancer
  useCertManager: true

$ tanzu package available list
- Retrieving package versions for
  NAME                      VERSION                RELEASED-AT  1.17.1+vmware.1-tkg.1  2021-07-23T18:00:00Z

$ tanzu package install contour -p --version 1.17.1+vmware.1-tkg.1 --values-file contour-simple.yaml
/ Installing package ''
| Getting namespace 'default'
| Getting package metadata for ''
| Creating service account 'contour-default-sa'
| Creating cluster admin role 'contour-default-cluster-role'
| Creating cluster role binding 'contour-default-cluster-rolebinding'
| Creating secret 'contour-default-values'
- Creating package resource
\ Package install status: Reconciling

 Added installed package 'contour' in namespace 'default'

$ tanzu package installed list
- Retrieving installed packages...
  NAME          PACKAGE-NAME                   PACKAGE-VERSION        STATUS
  cert-manager  1.1.0+vmware.1-tkg.2   Reconcile succeeded
  contour       1.17.1+vmware.1-tkg.1  Reconcile succeeded

$ kubectl get pods -A | grep 'contour\|envoy'
tanzu-system-ingress   contour-5dccf8dd5f-dqdbh                                   1/1     Running   0          93s
tanzu-system-ingress   contour-5dccf8dd5f-s5pvc                                   1/1     Running   0          93s
tanzu-system-ingress   envoy-q9nt8                                                2/2     Running   0          94s

$ kubectl get svc -A | grep envoy
tanzu-system-ingress   envoy                      LoadBalancer    xx.yy.112.159   80:30298/TCP,443:31028/TCP   104s

Cert Manager and Contour are now successfully deployed, and Envoy (part of Contour) has received a Load Balancer IP address (which I have partially obfuscated). We are good to proceed to the next steps.

Step 3 – Configure DNS

Next, add this VIP/LB IP Address of envoy to your DNS so that the application you plan to create (in my case, Kuard – Kubernetes Up and Running demo) resolves to it. For this demonstration, I am using the FDQN

$ nslookup

Non-authoritative answer:
Address: xx.yy.112.159

Step 4 – Setup a local Certificate Authority

Not everyone has access to a valid Certificate Authority. Thus, you can follow these steps to create your own local CA. There are 3 steps in total, as you will need to create a non-password-protected key that can be used by Kubernetes. For more details about creating a local CA, check out this very good blog from my colleague Jorge. Note that the default Ubuntu OpenSSL configuration had all of the necessary v3 extensions in place, so I did not need to change anything. This may not be the case on your particular distro, so again, refer to Jorge’s blog on how to do that if necessary.

  1. Create a private key: openssl genrsa -des3 -out localRootCA.key 2048
  2. Create the CA with SSLv3 extensions using the private key from the previous step: openssl req -x509 -new -nodes -key localRootCA.key -reqexts v3_req -extensions v3_ca -sha256 -days 1825 -out localRootCA.pem
  3. The command in step 2 created a password protected CA key. This command creates a password-unprotected version: openssl rsa -in localRootCA.key -out unprotected-localRootCA.key. A password unprotected version is needed for the Kubernetes secret in the next step.

Step 5 – Create a K8s secret with the CA info

In this step, we take the CA files created in step 4 and build a Kubernetes TLS secret from them.

$ kubectl create secret tls my-ca-secret --key unprotected-localRootCA.key --cert localRootCA.pem -n cert-manager
secret/my-ca-secret created

Step 6 – Deploy Kuard

We can now roll out the application. Here is a sample manifest to deploy the kuard application, along with what you should observe after it has been deployed. Note that this application has no external ip address associated with it.

$ cat kuard.yaml
apiVersion: apps/v1
kind: Deployment
  name: kuard
      app: kuard
  replicas: 1
        app: kuard
      - image:
        imagePullPolicy: Always
        name: kuard
        - containerPort: 8080
apiVersion: v1
kind: Service
  name: kuard
  - port: 80
    targetPort: 8080
    protocol: TCP
    app: kuard

$ kubectl apply -f kuard.yaml
deployment.apps/kuard created
service/kuard created

$ kubectl get deploy,svc
deployment.apps/kuard  1/1    1            1          27s

NAME                TYPE        CLUSTER-IP      EXTERNAL-IP  PORT(S)  AGE
service/kuard       ClusterIP  <none>        80/TCP    26s
service/kubernetes  ClusterIP      <none>        443/TCP  24m

Step 7 – Create a Certificate Issuer

Now we want to be able to access this application externally, but we also want to access it securely. We can do this via an Ingress. The first thing we need to do is create a ClusterIssuer for the certificates. Note that in the spec below is a reference to a Certificate Authority (CA) and the secret is the one we created earlier in step 5. We will make this a cluster-wide ClusterIssuer. Otherwise you will need to make an Issuer object on every namespace. Here is the manifest that you need to apply.

$ cat ca-issuer.yaml
kind: ClusterIssuer
  name: ca-issuer
  namespace: cert-manager
    secretName: my-ca-secret

$ kubectl apply -f ca-issuer.yaml created

Step 8 – Create the Ingress

In this step, we will create the Ingress with certificate annotations that point to our previously created ClusterIssuer. The Ingress class is Contour, and this is identified via another annotation. When there is a request to the kuard application, identified by its FQDN (, through the Ingress, a certificate will be automatically created by cert manager and stored in the secret myapp-cert. If everything is working correctly, the Ingress for our application should show the Envoy Load Balancer IP address.

$ cat ingress-with-cert.yaml
kind: Ingress
  name: kuard
  annotations: "contour" "ca-issuer"

      name: kuard
        number: 80
  - host:
      - path: /
        pathType: Prefix
            name: kuard
              number: 80
  - hosts:
    secretName: myapp-cert

$ kubectl apply -f ingress-with-cert.yaml created

$ kubectl get ingress
NAME    CLASS    HOSTS                  ADDRESS          PORTS    AGE
kuard   <none>  xx.yy.112.159   80, 443  4s

Step 9 – Verify everything is working

Now that the Ingress has been successfully created, we can now try to access our application via a browser, such as Firefox. If you tried to access this application through an Ingress that had no certificate integration, you will probably observe a security risk warning and a message about self-signed certificates – something like this:

Now with our own local CA in place for certificates, the message should change to something like this:

The reason for this error is that we created our own local CA, and the browser will not know who issued that certificate. If you used a recognized certificate authority, you should not observe this if the browser has the issuer in its list of CAs. What we can do to avoid this warning is import our CA into to the CA that the browser trusts.

Once the local CA has been added to the browser, I can now connect to my application without any errors.

And now we are successfully using the Cert Manager and Contour Carvel packages available in TKG to establish trust between our client (browser) and the application running in Kubernetes. Hope you find this useful.

The manifests used in this post can be found here on GitHub.