Tanzu Kubernetes with embedded Harbor Image Registry (revisited)
Just recently I had reason to have my TKG (Tanzu Kubernetes) guest cluster pull images from the embedded Harbor container image registry which is available as part of vSphere with Tanzu. Now, I did this in the past but there were quite a few hoops that you needed to jump through in order to make this work. I wrote about how I did it here. So I was pleased to see that the following update was included in the vSphere with Tanzu Release Notes that coincided with vSphere 7.0U1c last December:
Integration with Registry Service – Newly created Tanzu Kubernetes clusters work out of the box with
the vSphere Registry Service. Existing clusters, once updated to a new version, also work with the
Registry Service.
After recently upgrading my environment to VMware Cloud Foundation (VCF) v4.2, now was a good time to test out this new functionality. Let’s begin by showing the sorts of events thrown by a Pod, deployed in a TKG “guest” cluster, when it’s manifest has been configured to pull its image from the embedded Harbor image registry:
Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Pulling <invalid> (x3 over <invalid>) kubelet, tkg-cluster-vcf-w-tanzu-workers-dxdq6-75bc686795-hqtvt Pulling image "20.0.0.2/cormac-ns/cassandra:v11" Warning Failed <invalid> (x3 over <invalid>) kubelet, tkg-cluster-vcf-w-tanzu-workers-dxdq6-75bc686795-hqtvt Failed to pull image "20.0.0.2/cormac-ns/cassandra:v11": \ rpc error: code = Unknown desc = failed to pull and unpack image "20.0.0.2/cormac-ns/cassandra:v11": failed to resolve reference "20.0.0.2/cormac-ns/cassandra:v11": pull access denied, \ repository does not exist or may require authorization: server message: insufficient_scope: authorization failed Warning Failed <invalid> (x3 over <invalid>) kubelet, tkg-cluster-vcf-w-tanzu-workers-dxdq6-75bc686795-hqtvt Error: ErrImagePull Normal BackOff <invalid> (x3 over <invalid>) kubelet, tkg-cluster-vcf-w-tanzu-workers-dxdq6-75bc686795-hqtvt Back-off pulling image "20.0.0.2/cormac-ns/cassandra:v11" Warning Failed <invalid> (x3 over <invalid>) kubelet, tkg-cluster-vcf-w-tanzu-workers-dxdq6-75bc686795-hqtvt Error: ImagePullBackOff
The issue is that we do not have provided any credentials to the registry to do the pull, thus may require authorization. There are two steps we need to implement to address this issue:
- Create a secret that contains the Harbor image registry credentials
- Modify the default service account of the namespace where the Pod is being provisioned to reference the registry credentials secret
Let’s look at how to do those two steps in more detail:
Create a Registry Credential secret
The steps to create a secret for registry credentials is documented in the Kubernetes documentation. However we can add the steps here again for completion. To begin, we need credentials from a valid docker login to the embedded Harbor Image registry. This creates a ~/.docker/config.json file which holds credentials which can be used to create the secret that your TKG Pods can use to access the image registry.
Here is my ~/.docker/config.json:
$ cat ~/.docker/config.json
{
"auths": {
"20.0.0.2": {
"auth": "YWRtaW5pc3RyYXRvckB2c3BoZXJlLmxvY2FsOlZNd2FyZTEyMyE="
}
}
}
20.0.0.2 is the IP address of the embedded Harbor Image Registry. We now create a secret from this .json file. Remember that this secret needs to exist in the namespace where you are deploying the Pods. The namespace that I am using for this demonstration is called pull-test. So before I create the secret, I am changing contexts to that namespace.
$ kubectl config set-context --current --namespace pull-test Context "tkg-cluster-vcf-w-tanzu" modified. $ kubectl create secret generic regcred \ --from-file=.dockerconfigjson=/home/cormac/.docker/config.json \ --type=kubernetes.io/dockerconfigjson secret/regcred created $ kubectl get secret regcred -o yaml apiVersion: v1 data: .dockerconfigjson: ewoJImF1dGhzIjogewoJCSIyMC4wLjAuMiI6IHsKCQkJImF1dGgiOiAiWVdSdGFXNXBjM1J5WVhSdmNrQjJjM0JvWlhKbExteHZZMkZzT2xaTmQyRnlaVEV5TXlFPSIKCQl9LAoJCSJoYXJib3ItcmVwby52bXdhcmUuY29tIjogewoJCQkiYXV0aCI6ICJZMmh2WjJGdU9rSnNkV1ZRYjNCd2FXVTNNU0U9IgoJCX0sCgkJImh0dHBzOi8vaW5kZXguZG9ja2VyLmlvL3YxLyI6IHsKCQkJImF1dGgiOiAiWTI5eWJXRmphRzluWVc0NlNEUnljSE5rTTI0aCIKCQl9LAoJCSJpbmRleC5kb2NrZXIuaW8iOiB7CgkJCSJhdXRoIjogIlkyOXliV0ZqYUc5bllXNDZTRFJ5Y0hOa00yNGgiCgkJfSwKCQkicXVheS5pbyI6IHsKCQkJImF1dGgiOiAiWTI5eWJXRmphRzluWVc0NlNEUnljSE5rTTI0aCIKCQl9LAoJCSJyZWdpc3RyeS0xLmRvY2tlci5pbyI6IHsKCQkJImF1dGgiOiAiWTI5eWJXRmphRzluWVc0NlNEUnljSE5rTTI0aCIKCQl9Cgl9LAoJIkh0dHBIZWFkZXJzIjogewoJCSJVc2VyLUFnZW50IjogIkRvY2tlci1DbGllbnQvMTkuMDMuMTIgKGxpbnV4KSIKCX0KfQ== kind: Secret metadata: creationTimestamp: "2021-03-09T10:49:59Z" managedFields: - apiVersion: v1 fieldsType: FieldsV1 fieldsV1: f:data: .: {} f:.dockerconfigjson: {} f:type: {} manager: kubectl operation: Update time: "2021-03-09T10:49:59Z" name: regcred namespace: pull-test resourceVersion: "31256684" selfLink: /api/v1/namespaces/pull-test/secrets/regcred uid: 9a9880d8-2514-4bff-b12c-eb47220541c5 type: kubernetes.io/dockerconfigjson $ kubectl get secrets NAME TYPE DATA AGE default-token-ntcph kubernetes.io/service-account-token 3 19m regcred kubernetes.io/dockerconfigjson 1 90s
OK – now the regcred secret is now created with credentials to access to the embedded Harbor Image Registry. Now we need to add this secret to the default Service Account for the namespace.
Add secret to default Service Account for namespace
This procedure is also available in the official Kubernetes documentation. The steps are as follows:
- Export the current default Service Account in YAML format
- Modify the resulting YAML file to include the registration credentials secret
- Replace the current default Service Account with the new default Service Account
Let’s go through that, step by step.
$ kubectl get serviceaccounts NAME SECRETS AGE default 1 85d $ kubectl get serviceaccounts default -o yaml apiVersion: v1 kind: ServiceAccount metadata: creationTimestamp: "2021-03-09T10:32:00Z" name: default namespace: pull-test resourceVersion: "385" selfLink: /api/v1/namespaces/default/serviceaccounts/default uid: ebe51ce8-7971-4193-aa52-ceb42468913c secrets: - name: default-token-ntcph $ kubectl get serviceaccounts default -o yaml > sa.yaml
At this point, the Service Account information is now in the file sa.yaml. We now need to edit that file, removing the resourceVersion line, and adding an entry for the ImagePullSecrets to use our recently created regcred secret. It should look something like the following when editing is complete:
$ cat sa.yaml apiVersion: v1 kind: ServiceAccount metadata: creationTimestamp: "2021-03-09T10:32:00Z" name: default namespace: pull-test selfLink: /api/v1/namespaces/pull-test/serviceaccounts/default uid: ebe51ce8-7971-4193-aa52-ceb42468913c secrets: - name: default-token-ntcph imagePullSecrets: - name: regcred
Now we simply replace the current ServiceAccount for the namespace (in this case, it is called pull-test), and we can do one final check to ensure that the ImagePullSecret is now in place.
$ kubectl replace serviceaccount default -f ./sa.yaml serviceaccount/default replaced $ kubectl get serviceaccounts default -o yaml apiVersion: v1 imagePullSecrets: - name: regcred kind: ServiceAccount metadata: creationTimestamp: "2021-03-09T10:32:00Z" name: default namespace: pull-test resourceVersion: "31257588" selfLink: /api/v1/namespaces/pull-test/serviceaccounts/default uid: ebe51ce8-7971-4193-aa52-ceb42468913c secrets: - name: default-token-ntcph
Now any Pod images that are pulled from the embedded Harbor Image Registry in this namespace (pull-test) will utilize the registry credential contained in the regcred secret. If you need other TKG cluster namespaces to pull from the registry, simply repeat this process in that namespace.
solved the issue in my environment, thanks for the same