How to access embedded shared image registry from TKG cluster
vSphere with Tanzu ships with an embedded Harbor Image Registry to store container images. However, by default, TKG clusters deployed in a vSphere Namespace cannot access the registry. In this post, I will demonstrate how to allow a TKG guest / workload cluster to access the Harbor Image Registry. To do that, the image registry secret is retrieved at the vSphere Namespace level, and a new secret matching the Harbor Image Registry secret is created in the TKG cluster. Once created, this TKG level secret can be used to authenticate and pull container images for pods in the TKG cluster.
Step 1. Get the secret from the namespace context
In this first step, login to the vSphere Namespace context. This is the context where the TKG cluster is deployed. From here, retrieve the Harbor Image Registry secret in YAML format. Store it in a file by redirecting the output of the kubectl get secret command.
% kubectl-vsphere login --vsphere-username administrator@vsphere.local \ --server=https://xx.xx.xx.130 --insecure-skip-tls-verify Logged in successfully. You have access to the following contexts: xx.xx.xx.130 cormac-ns If the context you wish to use is not in this list, you may need to try logging in again later, or contact your cluster administrator. To change context, use `kubectl config use-context <workload name>` % kubectl config use-context cormac-ns Switched to context "cormac-ns". % kubectl get secret -n cormac-ns NAME TYPE DATA AGE cormac-ns-default-image-pull-secret kubernetes.io/dockerconfigjson 1 3d1h cormac-ns-default-image-push-secret kubernetes.io/dockerconfigjson 1 3d1h default-token-kjh97 kubernetes.io/service-account-token 3 3d22h tkg-cluster-v1-21-6-antrea kubernetes.io/tls 3 3d22h tkg-cluster-v1-21-6-auth-svc-cert kubernetes.io/tls 3 3d22h tkg-cluster-v1-21-6-ca Opaque 2 3d22h tkg-cluster-v1-21-6-ccm-token-f6d5r kubernetes.io/service-account-token 3 3d22h tkg-cluster-v1-21-6-control-plane-wlnj5 cluster.x-k8s.io/secret 1 3d1h tkg-cluster-v1-21-6-encryption Opaque 1 3d22h tkg-cluster-v1-21-6-etcd Opaque 2 3d22h tkg-cluster-v1-21-6-extensions-ca kubernetes.io/tls 3 3d22h tkg-cluster-v1-21-6-kubeconfig Opaque 1 3d22h tkg-cluster-v1-21-6-metrics-server-cert kubernetes.io/tls 3 3d22h tkg-cluster-v1-21-6-proxy Opaque 2 3d22h tkg-cluster-v1-21-6-pvcsi-token-2d5rf kubernetes.io/service-account-token 3 3d22h tkg-cluster-v1-21-6-sa Opaque 2 3d22h tkg-cluster-v1-21-6-ssh kubernetes.io/ssh-auth 1 3d22h tkg-cluster-v1-21-6-ssh-password Opaque 1 3d22h tkg-cluster-v1-21-6-workers-sjbkl-44v79 cluster.x-k8s.io/secret 1 3d1h tkg-cluster-v1-21-6-workers-sjbkl-wd7xl cluster.x-k8s.io/secret 1 3d1h % kubectl get secret cormac-ns-default-image-pull-secret NAME TYPE DATA AGE cormac-ns-default-image-pull-secret kubernetes.io/dockerconfigjson 1 3d1h % kubectl get secret cormac-ns-default-image-pull-secret -o yaml apiVersion: v1 data: .dockerconfigjson: ewoJCxxxQkJfQ== kind: Secret metadata: creationTimestamp: "2022-05-06T08:24:51Z" name: cormac-ns-default-image-pull-secret namespace: cormac-ns ownerReferences: - apiVersion: registryagent.vmware.com/v1alpha1 blockOwnerDeletion: true controller: true kind: Project name: cormac-ns uid: 4e89eb73-bf14-4ac9-b536-205b0f7923ae resourceVersion: "8090402" selfLink: /api/v1/namespaces/cormac-ns/secrets/cormac-ns-default-image-pull-secret uid: 393395c0-0321-4b2f-af90-bbc00d0bcbe0 type: kubernetes.io/dockerconfigjson % kubectl get secret cormac-ns-default-image-pull-secret -o yaml > image-pull-secret.yaml %
Step 2. Edit the secret for the TKG cluster context
The secret is stored in a file as seen previously. Modify the file contents so that the secret can be successfully applied to the TKG workload cluster. This means the namespace and the name entries should be modified. The rest of the fields can be left at their default values. In this example, the secret is created in the default namespace in the TKG cluster, and it is renamed to harbor-registry-secret, as shown below.
apiVersion: v1 data: .dockerconfigjson: ewoJCxxxQkJfQ== kind: Secret metadata: creationTimestamp: "2022-05-06T08:24:51Z" name: harbor-registry-secret namespace: default ownerReferences: - apiVersion: registryagent.vmware.com/v1alpha1 blockOwnerDeletion: true controller: true kind: Project name: cormac-ns uid: 4e89eb73-bf14-4ac9-b536-205b0f7923ae resourceVersion: "8090402" selfLink: /api/v1/namespaces/cormac-ns/secrets/cormac-ns-default-image-pull-secret uid: 393395c0-0321-4b2f-af90-bbc00d0bcbe0 type: kubernetes.io/dockerconfigjson
Step 3. Switch contexts to the TKG cluster, and apply the new secret
Switch contexts from the vSphere Namespace to the context of the TKG guest / workload cluster. First, logout of the current context, and then log back in using the TKG context shown below. Then apply the modified secret YAML manifest that we retrieved from the vSphere Namespace context above.
% kubectl-vsphere logout Logged out of all vSphere namespaces. % kubectl-vsphere login --vsphere-username administrator@vsphere.local \ --server=https://xx.xx.xx.130 --insecure-skip-tls-verify --tanzu-kubernetes-cluster-namespace \ cormac-ns --tanzu-kubernetes-cluster-name tkg-cluster-v1-21-6 Logged in successfully. You have access to the following contexts: xx.xx.xx.130 cormac-ns tkg-cluster-v1-21-6 If the context you wish to use is not in this list, you may need to try logging in again later, or contact your cluster administrator. To change context, use `kubectl config use-context <workload name>` % kubectl config use-context tkg-cluster-v1-21-6 Switched to context "tkg-cluster-v1-21-6". % kubectl get nodes NAME STATUS ROLES AGE VERSION tkg-cluster-v1-21-6-control-plane-9fn5c Ready control-plane,master 3d1h v1.21.6+vmware.1 tkg-cluster-v1-21-6-worker-pool-1-x2szc-7ffbddd567-8ct8m Ready <none> 3d1h v1.21.6+vmware.1 tkg-cluster-v1-21-6-worker-pool-1-x2szc-7ffbddd567-pn7sp Ready <none> 3d1h v1.21.6+vmware.1 % kubectl get ns NAME STATUS AGE cert-manager Active 3d21h default Active 3d22h kube-node-lease Active 3d22h kube-public Active 3d22h kube-system Active 3d22h tanzu-package-repo-global Active 3d21h tanzu-system-dashboard Active 3d20h tanzu-system-ingress Active 3d21h tanzu-system-monitoring Active 3d21h tanzu-system-service-discovery Active 3d21h tkg-system Active 3d21h vmware-system-auth Active 3d22h vmware-system-cloud-provider Active 3d22h vmware-system-csi Active 3d22h % kubectl apply -f image-pull-secret.yaml secret/harbor-registry-secret created % kubectl get secret NAME TYPE DATA AGE alertmanager-overlay Opaque 1 3d19h cert-manager-default-sa-token-z7c7q kubernetes.io/service-account-token 3 3d21h contour-default-sa-token-d5twq kubernetes.io/service-account-token 3 3d21h contour-default-values Opaque 1 3d21h default-token-rkmjd kubernetes.io/service-account-token 3 3d22h external-dns-default-sa-token-78f7j kubernetes.io/service-account-token 3 3d21h external-dns-default-values Opaque 1 3d21h grafana-default-sa-token-8t499 kubernetes.io/service-account-token 3 3d20h grafana-default-values Opaque 1 3d20h harbor-registry-secret kubernetes.io/dockerconfigjson 1 13s prometheus-default-sa-token-5srk8 kubernetes.io/service-account-token 3 3d21h prometheus-default-values Opaque 1 3d21h
Step 4. Retrieve images from embedded Harbor in TKG
The Harbor Image Registry secret in the TKG cluster has been created It should now be possible to pull images from the embedded Harbor Image Registry. Let’s try that with a simple busybox pod. Note that the image is being pulled from the embedded Harbor Image Registry. The imagePullSecret references our newly created secret which contains the relevant information to successfully connect to the registry.
% cat busybox-pod-HRBR.yaml apiVersion: v1 kind: Pod metadata: name: busybox1 namespace: default labels: app: busybox1 spec: containers: - name: busybox image: xx.xx.xx.134/cormac-ns/busybox:latest command: - sleep - "3600" imagePullPolicy: IfNotPresent imagePullSecrets: - name: harbor-registry-secret restartPolicy: Always % kubectl apply -f busybox-pod-HRBR.yaml pod/busybox1 created % kubectl get pods NAME READY STATUS RESTARTS AGE busybox1 1/1 Running 0 4s
Success! Containers in pods are now able to pull images from the embedded Harbor Image Registry from within the TKG guest/workload cluster.
Note that the embedded Harbor Image Registry has a dependency on NSX-T at the time of writing (vSphere with Tanzu 7.0.3). vSphere with Tanzu with NSX-T networking is available for both on-premises deployments and cloud deployments. The latter is available via Tanzu Services on VMware Cloud on AWS.
