Tanzu Kubernetes with embedded Harbor Image Registry (revisited)

Just recently I had reason to have my TKG (Tanzu Kubernetes) guest cluster pull images from the embedded Harbor container image registry which is available as part of vSphere with Tanzu. Now, I did this in the past but there were quite a few hoops that you needed to jump through in order to make this work. I wrote about how I did it here. So I was pleased to see that the following update was included in the vSphere with Tanzu Release Notes that coincided with vSphere 7.0U1c last December:

Integration with Registry Service – Newly created Tanzu Kubernetes clusters work out of the box with 
the vSphere Registry Service. Existing clusters, once updated to a new version, also work with the 
Registry Service.

After recently upgrading my environment to VMware Cloud Foundation (VCF) v4.2, now was a good time to test out this new functionality. Let’s begin by showing the sorts of events thrown by a Pod, deployed in a TKG “guest” cluster, when it’s manifest has been configured to pull its image from the embedded Harbor image registry:

Events:

  Type     Reason                  Age                            From                                                             Message
  ----     ------                  ----                           ----                                                             -------
  Normal   Pulling                 <invalid> (x3 over <invalid>)  kubelet, tkg-cluster-vcf-w-tanzu-workers-dxdq6-75bc686795-hqtvt  Pulling image "20.0.0.2/cormac-ns/cassandra:v11"
  Warning  Failed                  <invalid> (x3 over <invalid>)  kubelet, tkg-cluster-vcf-w-tanzu-workers-dxdq6-75bc686795-hqtvt  Failed to pull image "20.0.0.2/cormac-ns/cassandra:v11": \
rpc error: code = Unknown desc = failed to pull and unpack image "20.0.0.2/cormac-ns/cassandra:v11": failed to resolve reference "20.0.0.2/cormac-ns/cassandra:v11": pull access denied, \
repository does not exist or may require authorization: server message: insufficient_scope: authorization failed
  Warning  Failed                  <invalid> (x3 over <invalid>)  kubelet, tkg-cluster-vcf-w-tanzu-workers-dxdq6-75bc686795-hqtvt  Error: ErrImagePull
  Normal   BackOff                 <invalid> (x3 over <invalid>)  kubelet, tkg-cluster-vcf-w-tanzu-workers-dxdq6-75bc686795-hqtvt  Back-off pulling image "20.0.0.2/cormac-ns/cassandra:v11"
  Warning  Failed                  <invalid> (x3 over <invalid>)  kubelet, tkg-cluster-vcf-w-tanzu-workers-dxdq6-75bc686795-hqtvt  Error: ImagePullBackOff

The issue is that we do not have provided any credentials to the registry to do the pull, thus may require authorization. There are two steps we need to implement to address this issue:

  1. Create a secret that contains the Harbor image registry credentials
  2. Modify the default service account of the namespace where the Pod is being provisioned to reference the registry credentials secret

Let’s look at how to do those two steps in more detail:

Create a Registry Credential secret

The steps to create a secret for registry credentials is documented in the Kubernetes documentation. However we can add the steps here again for completion. To begin, we need credentials from a valid docker login to the embedded Harbor Image registry. This creates a ~/.docker/config.json file which holds credentials which can be used to create the secret that your TKG Pods can use to access the image registry.

Here is my ~/.docker/config.json:

$ cat ~/.docker/config.json
{
        "auths": {
                "20.0.0.2": {
                        "auth": "YWRtaW5pc3RyYXRvckB2c3BoZXJlLmxvY2FsOlZNd2FyZTEyMyE="
                }
        }
}

20.0.0.2 is the IP address of the embedded Harbor Image Registry. We now create a secret from this .json file. Remember that this secret needs to exist in the namespace where you are deploying the Pods. The namespace that I am using for this demonstration is called pull-test. So before I create the secret, I am changing contexts to that namespace.

$ kubectl config set-context --current --namespace pull-test
Context "tkg-cluster-vcf-w-tanzu" modified.


$ kubectl create secret generic regcred \
--from-file=.dockerconfigjson=/home/cormac/.docker/config.json \
--type=kubernetes.io/dockerconfigjson
secret/regcred created


$ kubectl get secret regcred -o yaml
apiVersion: v1
data:
  .dockerconfigjson: ewoJImF1dGhzIjogewoJCSIyMC4wLjAuMiI6IHsKCQkJImF1dGgiOiAiWVdSdGFXNXBjM1J5WVhSdmNrQjJjM0JvWlhKbExteHZZMkZzT2xaTmQyRnlaVEV5TXlFPSIKCQl9LAoJCSJoYXJib3ItcmVwby52bXdhcmUuY29tIjogewoJCQkiYXV0aCI6ICJZMmh2WjJGdU9rSnNkV1ZRYjNCd2FXVTNNU0U9IgoJCX0sCgkJImh0dHBzOi8vaW5kZXguZG9ja2VyLmlvL3YxLyI6IHsKCQkJImF1dGgiOiAiWTI5eWJXRmphRzluWVc0NlNEUnljSE5rTTI0aCIKCQl9LAoJCSJpbmRleC5kb2NrZXIuaW8iOiB7CgkJCSJhdXRoIjogIlkyOXliV0ZqYUc5bllXNDZTRFJ5Y0hOa00yNGgiCgkJfSwKCQkicXVheS5pbyI6IHsKCQkJImF1dGgiOiAiWTI5eWJXRmphRzluWVc0NlNEUnljSE5rTTI0aCIKCQl9LAoJCSJyZWdpc3RyeS0xLmRvY2tlci5pbyI6IHsKCQkJImF1dGgiOiAiWTI5eWJXRmphRzluWVc0NlNEUnljSE5rTTI0aCIKCQl9Cgl9LAoJIkh0dHBIZWFkZXJzIjogewoJCSJVc2VyLUFnZW50IjogIkRvY2tlci1DbGllbnQvMTkuMDMuMTIgKGxpbnV4KSIKCX0KfQ==
kind: Secret
metadata:
  creationTimestamp: "2021-03-09T10:49:59Z"
  managedFields:
  - apiVersion: v1
    fieldsType: FieldsV1
    fieldsV1:
      f:data:
        .: {}
        f:.dockerconfigjson: {}
      f:type: {}
    manager: kubectl
    operation: Update
    time: "2021-03-09T10:49:59Z"
  name: regcred
  namespace: pull-test
  resourceVersion: "31256684"
  selfLink: /api/v1/namespaces/pull-test/secrets/regcred
  uid: 9a9880d8-2514-4bff-b12c-eb47220541c5
type: kubernetes.io/dockerconfigjson


$ kubectl get secrets
NAME                  TYPE                                  DATA   AGE
default-token-ntcph   kubernetes.io/service-account-token   3      19m
regcred               kubernetes.io/dockerconfigjson        1      90s

OK – now the regcred secret is now created with credentials to access to the embedded Harbor Image Registry. Now we need to add this secret to the default Service Account for the namespace.

Add secret to default Service Account for namespace

This procedure is also available in the official Kubernetes documentation. The steps are as follows:

  1. Export the current default Service Account in YAML format
  2. Modify the resulting YAML file to include the registration credentials secret
  3. Replace the current default Service Account with the new default Service Account

Let’s go through that, step by step.

$ kubectl get serviceaccounts
NAME      SECRETS   AGE
default   1         85d


$ kubectl get serviceaccounts default -o yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  creationTimestamp: "2021-03-09T10:32:00Z"
  name: default
  namespace: pull-test
  resourceVersion: "385"
  selfLink: /api/v1/namespaces/default/serviceaccounts/default
  uid: ebe51ce8-7971-4193-aa52-ceb42468913c
secrets:
- name: default-token-ntcph


$ kubectl get serviceaccounts default -o yaml > sa.yaml

At this point, the Service Account information is now in the file sa.yaml. We now need to edit that file, removing the resourceVersion line, and adding an entry for the ImagePullSecrets to use our recently created regcred secret. It should look something like the following when editing is complete:

$ cat sa.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  creationTimestamp: "2021-03-09T10:32:00Z"
  name: default
  namespace: pull-test
  selfLink: /api/v1/namespaces/pull-test/serviceaccounts/default
  uid: ebe51ce8-7971-4193-aa52-ceb42468913c
secrets:
- name: default-token-ntcph
imagePullSecrets:
- name: regcred

Now we simply replace the current ServiceAccount for the namespace (in this case, it is called pull-test), and we can do one final check to ensure that the ImagePullSecret is now in place.

$ kubectl replace serviceaccount default -f ./sa.yaml
serviceaccount/default replaced


$ kubectl get serviceaccounts default -o yaml
apiVersion: v1
imagePullSecrets:
- name: regcred
kind: ServiceAccount
metadata:
  creationTimestamp: "2021-03-09T10:32:00Z"
  name: default
  namespace: pull-test
  resourceVersion: "31257588"
  selfLink: /api/v1/namespaces/pull-test/serviceaccounts/default
  uid: ebe51ce8-7971-4193-aa52-ceb42468913c
secrets:
- name: default-token-ntcph

Now any Pod images that are pulled from the embedded Harbor Image Registry in this namespace (pull-test) will utilize the registry credential contained in the regcred secret. If you need other TKG cluster namespaces to pull from the registry, simply repeat this process in that namespace.

One Reply to “Tanzu Kubernetes with embedded Harbor Image Registry (revisited)”

Comments are closed.