Securing application Ingress access on TKG v1.4 with Cert Manager and Contour

In this article, I will walk through the steps involved in securing application Ingress access on TKG v1.4. To achieve this, I will use 2 packages that are available with TKG v1.4, Cert Manager and Contour. We will deploy a sample application kuard – Kubernetes Up and Running demo, and show how we can use these packages to automatically generated certificates to establish trust between our client (browser) and the application (kuard) which will be accessed via an Ingress. For the purposes of this article, I will create my own local Certificate Authority. If you have access to a valid CA, you might prefer to use that instead. At the end of this procedure, we should see how a user can point their browser to the FQDN of the application, and connect to it without the browser reporting any security warnings.

Note that there are some requirements before we begin. The assumption is that you have already deployed a TKG v1.4 workload cluster to work with, along with the Carvel Tools, and have kubectl available to query the cluster. A Load Balancer service is also required. This Tanzu Kubernetes Grid deployment is on vSphere so I am using the NSX Advanced Load Balancer in this example. Note that I am doing all of these operations from an Ubuntu 20.04.2 distro. Some commands, such as openssl, might be subtly different if you are on another distro.

Step 1 – Install Cert Manager

The first step is to install the Cert Manager package onto the workload cluster. No bespoke parameters or changes are needed. We can simply deploy the default.

$ tanzu package available get cert-manager.tanzu.vmware.com
- Retrieving package details for cert-manager.tanzu.vmware.com...
NAME:                 cert-manager.tanzu.vmware.com
DISPLAY-NAME:         cert-manager
SHORT-DESCRIPTION:    Certificate management
PACKAGE-PROVIDER:     VMware
LONG-DESCRIPTION:     Provides certificate management provisioning within the cluster
MAINTAINERS:          [{Nicholas Seemiller}]
SUPPORT:              Support provided by VMware for deployment on TKG 1.4+ clusters. Best-effort support for deployment on any conformant Kubernetes cluster. Contact support by opening a support request via VMware Cloud Services or my.vmware.com.
CATEGORY:             [certificate management]

$ tanzu package available list cert-manager.tanzu.vmware.com
- Retrieving package versions for cert-manager.tanzu.vmware.com...
  NAME                           VERSION               RELEASED-AT
  cert-manager.tanzu.vmware.com  1.1.0+vmware.1-tkg.2  2020-11-24T18:00:00Z

$ tanzu package install cert-manager --package-name  cert-manager.tanzu.vmware.com --version 1.1.0+vmware.1-tkg.2
/ Installing package 'cert-manager.tanzu.vmware.com'
| Getting namespace 'default'
| Getting package metadata for 'cert-manager.tanzu.vmware.com'
| Creating service account 'cert-manager-default-sa'
| Creating cluster admin role 'cert-manager-default-cluster-role'
| Creating cluster role binding 'cert-manager-default-cluster-rolebinding'
- Creating package resource
| Package install status: Reconciling

 Added installed package 'cert-manager' in namespace 'default'

$ tanzu package installed list
/ Retrieving installed packages...
  NAME          PACKAGE-NAME                   PACKAGE-VERSION       STATUS
  cert-manager  cert-manager.tanzu.vmware.com  1.1.0+vmware.1-tkg.2  Reconcile succeeded


$ kubectl get all -n cert-manager
NAME                                         READY   STATUS    RESTARTS   AGE
pod/cert-manager-8576785984-2rtbm            1/1     Running   0          119s
pod/cert-manager-cainjector-dbdb9546-52v66   1/1     Running   0          119s
pod/cert-manager-webhook-5f688f647b-spxqx    1/1     Running   0          119s

NAME                           TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
service/cert-manager           ClusterIP   100.66.243.200   <none>        9402/TCP   119s
service/cert-manager-webhook   ClusterIP   100.70.82.204    <none>        443/TCP    119s

NAME                                      READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/cert-manager              1/1     1            1           119s
deployment.apps/cert-manager-cainjector   1/1     1            1           119s
deployment.apps/cert-manager-webhook      1/1     1            1           119s

NAME                                               DESIRED   CURRENT   READY   AGE
replicaset.apps/cert-manager-8576785984            1         1         1       119s
replicaset.apps/cert-manager-cainjector-dbdb9546   1         1         1       119s
replicaset.apps/cert-manager-webhook-5f688f647b    1         1         1       119s

Step 2 – Install Contour

Contour is an open-source Ingress controller from VMware. We need to provide a small change to its default configuration so that it uses a Load Balancer service (provide by NSX ALB). We do this by passing a values manifest file at deployment time, as shown below.

$ cat contour-simple.yaml
envoy:
  service:
    type: LoadBalancer
certificates:
  useCertManager: true

$ tanzu package available list contour.tanzu.vmware.com
- Retrieving package versions for contour.tanzu.vmware.com...
  NAME                      VERSION                RELEASED-AT
  contour.tanzu.vmware.com  1.17.1+vmware.1-tkg.1  2021-07-23T18:00:00Z

$ tanzu package install contour -p contour.tanzu.vmware.com --version 1.17.1+vmware.1-tkg.1 --values-file contour-simple.yaml
/ Installing package 'contour.tanzu.vmware.com'
| Getting namespace 'default'
| Getting package metadata for 'contour.tanzu.vmware.com'
| Creating service account 'contour-default-sa'
| Creating cluster admin role 'contour-default-cluster-role'
| Creating cluster role binding 'contour-default-cluster-rolebinding'
| Creating secret 'contour-default-values'
- Creating package resource
\ Package install status: Reconciling

 Added installed package 'contour' in namespace 'default'

$ tanzu package installed list
- Retrieving installed packages...
  NAME          PACKAGE-NAME                   PACKAGE-VERSION        STATUS
  cert-manager  cert-manager.tanzu.vmware.com  1.1.0+vmware.1-tkg.2   Reconcile succeeded
  contour       contour.tanzu.vmware.com       1.17.1+vmware.1-tkg.1  Reconcile succeeded

$ kubectl get pods -A | grep 'contour\|envoy'
tanzu-system-ingress   contour-5dccf8dd5f-dqdbh                                   1/1     Running   0          93s
tanzu-system-ingress   contour-5dccf8dd5f-s5pvc                                   1/1     Running   0          93s
tanzu-system-ingress   envoy-q9nt8                                                2/2     Running   0          94s

$ kubectl get svc -A | grep envoy
tanzu-system-ingress   envoy                      LoadBalancer   100.68.55.154    xx.yy.112.159   80:30298/TCP,443:31028/TCP   104s

Cert Manager and Contour are now successfully deployed, and Envoy (part of Contour) has received a Load Balancer IP address (which I have partially obfuscated). We are good to proceed to the next steps.

Step 3 – Configure DNS

Next, add this VIP/LB IP Address of envoy to your DNS so that the application you plan to create (in my case, Kuard – Kubernetes Up and Running demo) resolves to it. For this demonstration, I am using the FDQN kuard.corinternal.com.

$ nslookup kuard.corinternal.com
Server:     127.0.0.53
Address:    127.0.0.53#53

Non-authoritative answer:
Name:   kuard.corinternal.com
Address: xx.yy.112.159

Step 4 – Setup a local Certificate Authority

Not everyone has access to a valid Certificate Authority. Thus, you can follow these steps to create your own local CA. There are 3 steps in total, as you will need to create a non-password-protected key that can be used by Kubernetes. For more details about creating a local CA, check out this very good blog from my colleague Jorge. Note that the default Ubuntu OpenSSL configuration had all of the necessary v3 extensions in place, so I did not need to change anything. This may not be the case on your particular distro, so again, refer to Jorge’s blog on how to do that if necessary.

  1. Create a private key: openssl genrsa -des3 -out localRootCA.key 2048
  2. Create the CA with SSLv3 extensions using the private key from the previous step: openssl req -x509 -new -nodes -key localRootCA.key -reqexts v3_req -extensions v3_ca -sha256 -days 1825 -out localRootCA.pem
  3. The command in step 2 created a password protected CA key. This command creates a password-unprotected version: openssl rsa -in localRootCA.key -out unprotected-localRootCA.key. A password unprotected version is needed for the Kubernetes secret in the next step.

Step 5 – Create a K8s secret with the CA info

In this step, we take the CA files created in step 4 and build a Kubernetes TLS secret from them.

$ kubectl create secret tls my-ca-secret --key unprotected-localRootCA.key --cert localRootCA.pem -n cert-manager
secret/my-ca-secret created

Step 6 – Deploy Kuard

We can now roll out the application. Here is a sample manifest to deploy the kuard application, along with what you should observe after it has been deployed. Note that this application has no external ip address associated with it.

$ cat kuard.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: kuard
spec:
  selector:
    matchLabels:
      app: kuard
  replicas: 1
  template:
    metadata:
      labels:
        app: kuard
    spec:
      containers:
      - image: gcr.io/kuar-demo/kuard-amd64:1
        imagePullPolicy: Always
        name: kuard
        ports:
        - containerPort: 8080
---
apiVersion: v1
kind: Service
metadata:
  name: kuard
spec:
  ports:
  - port: 80
    targetPort: 8080
    protocol: TCP
  selector:
    app: kuard


$ kubectl apply -f kuard.yaml
deployment.apps/kuard created
service/kuard created

$ kubectl get deploy,svc
NAME                    READY  UP-TO-DATE  AVAILABLE  AGE
deployment.apps/kuard  1/1    1            1          27s

NAME                TYPE        CLUSTER-IP      EXTERNAL-IP  PORT(S)  AGE
service/kuard       ClusterIP   100.70.248.113  <none>        80/TCP    26s
service/kubernetes  ClusterIP   100.64.0.1      <none>        443/TCP  24m

Step 7 – Create a Certificate Issuer

Now we want to be able to access this application externally, but we also want to access it securely. We can do this via an Ingress. The first thing we need to do is create a ClusterIssuer for the certificates. Note that in the spec below is a reference to a Certificate Authority (CA) and the secret is the one we created earlier in step 5. We will make this a cluster-wide ClusterIssuer. Otherwise you will need to make an Issuer object on every namespace. Here is the manifest that you need to apply.

$ cat ca-issuer.yaml
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: ca-issuer
  namespace: cert-manager
spec:
  ca:
    secretName: my-ca-secret


$ kubectl apply -f ca-issuer.yaml
clusterissuer.cert-manager.io/ca-issuer created

Step 8 – Create the Ingress

In this step, we will create the Ingress with certificate annotations that point to our previously created ClusterIssuer. The Ingress class is Contour, and this is identified via another annotation. When there is a request to the kuard application, identified by its FQDN (kuard.corinternal.com), through the Ingress, a certificate will be automatically created by cert manager and stored in the secret myapp-cert. If everything is working correctly, the Ingress for our application should show the Envoy Load Balancer IP address.

$ cat ingress-with-cert.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: kuard
  annotations:
    kubernetes.io/ingress.class: "contour"
    cert-manager.io/cluster-issuer: "ca-issuer"

spec:
  defaultBackend:
    service:
      name: kuard
      port:
        number: 80
  rules:
  - host: kuard.corinternal.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: kuard
            port:
              number: 80
  tls:
  - hosts:
    - kuard.corinternal.com
    secretName: myapp-cert

$ kubectl apply -f ingress-with-cert.yaml
ingress.networking.k8s.io/kuard created

$ kubectl get ingress
NAME    CLASS    HOSTS                  ADDRESS          PORTS    AGE
kuard   <none>   kuard.corinternal.com  xx.yy.112.159   80, 443  4s

Step 9 – Verify everything is working

Now that the Ingress has been successfully created, we can now try to access our application via a browser, such as Firefox. If you tried to access this application through an Ingress that had no certificate integration, you will probably observe a security risk warning and a message about self-signed certificates – something like this:

Now with our own local CA in place for certificates, the message should change to something like this:

The reason for this error is that we created our own local CA, and the browser will not know who issued that certificate. If you used a recognized certificate authority, you should not observe this if the browser has the issuer in its list of CAs. What we can do to avoid this warning is import our CA into to the CA that the browser trusts.

Once the local CA has been added to the browser, I can now connect to my application without any errors.

And now we are successfully using the Cert Manager and Contour Carvel packages available in TKG to establish trust between our client (browser) and the application running in Kubernetes. Hope you find this useful.

The manifests used in this post can be found here on GitHub.

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.