How to access embedded shared image registry from TKG cluster

vSphere with Tanzu ships with an embedded Harbor Image Registry to store container images. However, by default, TKG clusters deployed in a vSphere Namespace cannot access the registry. In this post, I will demonstrate how to allow a TKG guest / workload cluster to access the Harbor Image Registry. To do that, the image registry secret is retrieved at the vSphere Namespace level, and a new secret matching the Harbor Image Registry secret is created in the TKG cluster. Once created, this TKG level secret can be used to authenticate and pull container images for pods in the TKG cluster.

Step 1. Get the secret from the namespace context

In this first step, login to the vSphere Namespace context. This is the context where the TKG cluster is deployed. From here, retrieve the Harbor Image Registry secret in YAML format. Store it in a file by redirecting the output of the kubectl get secret command.

% kubectl-vsphere login --vsphere-username administrator@vsphere.local \
--server=https://xx.xx.xx.130 --insecure-skip-tls-verify

Logged in successfully.

You have access to the following contexts:
  xx.xx.xx.130
  cormac-ns

If the context you wish to use is not in this list, you may need to try
logging in again later, or contact your cluster administrator.

To change context, use `kubectl config use-context <workload name>`


% kubectl config use-context cormac-ns
Switched to context "cormac-ns".


% kubectl get secret -n cormac-ns
NAME                                      TYPE                                  DATA   AGE
cormac-ns-default-image-pull-secret       kubernetes.io/dockerconfigjson        1      3d1h
cormac-ns-default-image-push-secret       kubernetes.io/dockerconfigjson        1      3d1h
default-token-kjh97                       kubernetes.io/service-account-token   3      3d22h
tkg-cluster-v1-21-6-antrea                kubernetes.io/tls                     3      3d22h
tkg-cluster-v1-21-6-auth-svc-cert         kubernetes.io/tls                     3      3d22h
tkg-cluster-v1-21-6-ca                    Opaque                                2      3d22h
tkg-cluster-v1-21-6-ccm-token-f6d5r       kubernetes.io/service-account-token   3      3d22h
tkg-cluster-v1-21-6-control-plane-wlnj5   cluster.x-k8s.io/secret               1      3d1h
tkg-cluster-v1-21-6-encryption            Opaque                                1      3d22h
tkg-cluster-v1-21-6-etcd                  Opaque                                2      3d22h
tkg-cluster-v1-21-6-extensions-ca         kubernetes.io/tls                     3      3d22h
tkg-cluster-v1-21-6-kubeconfig            Opaque                                1      3d22h
tkg-cluster-v1-21-6-metrics-server-cert   kubernetes.io/tls                     3      3d22h
tkg-cluster-v1-21-6-proxy                 Opaque                                2      3d22h
tkg-cluster-v1-21-6-pvcsi-token-2d5rf     kubernetes.io/service-account-token   3      3d22h
tkg-cluster-v1-21-6-sa                    Opaque                                2      3d22h
tkg-cluster-v1-21-6-ssh                   kubernetes.io/ssh-auth                1      3d22h
tkg-cluster-v1-21-6-ssh-password          Opaque                                1      3d22h
tkg-cluster-v1-21-6-workers-sjbkl-44v79   cluster.x-k8s.io/secret               1      3d1h
tkg-cluster-v1-21-6-workers-sjbkl-wd7xl   cluster.x-k8s.io/secret               1      3d1h


% kubectl get secret cormac-ns-default-image-pull-secret
NAME                                  TYPE                            DATA  AGE
cormac-ns-default-image-pull-secret  kubernetes.io/dockerconfigjson  1      3d1h


% kubectl get secret cormac-ns-default-image-pull-secret -o yaml
apiVersion: v1
data:
  .dockerconfigjson: ewoJCxxxQkJfQ==
kind: Secret
metadata:
  creationTimestamp: "2022-05-06T08:24:51Z"
  name: cormac-ns-default-image-pull-secret
  namespace: cormac-ns
  ownerReferences:
  - apiVersion: registryagent.vmware.com/v1alpha1
    blockOwnerDeletion: true
    controller: true
    kind: Project
    name: cormac-ns
    uid: 4e89eb73-bf14-4ac9-b536-205b0f7923ae
  resourceVersion: "8090402"
  selfLink: /api/v1/namespaces/cormac-ns/secrets/cormac-ns-default-image-pull-secret
  uid: 393395c0-0321-4b2f-af90-bbc00d0bcbe0
type: kubernetes.io/dockerconfigjson


% kubectl get secret cormac-ns-default-image-pull-secret -o yaml > image-pull-secret.yaml
%

Step 2. Edit the secret for the TKG cluster context

The secret is stored in a file as seen previously. Modify the file contents so that the secret can be successfully applied to the TKG workload cluster. This means the namespace and the name entries should be modified. The rest of the fields can be left at their default values. In this example, the secret is created in the default namespace in the TKG cluster, and it is renamed to harbor-registry-secret, as shown below.

apiVersion: v1
data:
 .dockerconfigjson: ewoJCxxxQkJfQ==
kind: Secret
metadata:
 creationTimestamp: "2022-05-06T08:24:51Z"
 name: harbor-registry-secret
 namespace: default
 ownerReferences:
 - apiVersion: registryagent.vmware.com/v1alpha1
   blockOwnerDeletion: true
   controller: true
   kind: Project
   name: cormac-ns
   uid: 4e89eb73-bf14-4ac9-b536-205b0f7923ae
 resourceVersion: "8090402"
 selfLink: /api/v1/namespaces/cormac-ns/secrets/cormac-ns-default-image-pull-secret
 uid: 393395c0-0321-4b2f-af90-bbc00d0bcbe0
type: kubernetes.io/dockerconfigjson

Step 3. Switch contexts to the TKG cluster, and apply the new secret

Switch contexts from the vSphere Namespace to the context of the TKG guest / workload cluster. First, logout of the current context, and then log back in using the TKG context shown below. Then apply the modified secret YAML manifest that we retrieved from the vSphere Namespace context above.

% kubectl-vsphere logout
Logged out of all vSphere namespaces.


% kubectl-vsphere login --vsphere-username administrator@vsphere.local \
--server=https://xx.xx.xx.130 --insecure-skip-tls-verify --tanzu-kubernetes-cluster-namespace \
cormac-ns --tanzu-kubernetes-cluster-name tkg-cluster-v1-21-6

Logged in successfully.

You have access to the following contexts:
  xx.xx.xx.130
  cormac-ns
  tkg-cluster-v1-21-6

If the context you wish to use is not in this list, you may need to try
logging in again later, or contact your cluster administrator.

To change context, use `kubectl config use-context <workload name>`


% kubectl config use-context tkg-cluster-v1-21-6
Switched to context "tkg-cluster-v1-21-6".


% kubectl get nodes
NAME                                                      STATUS  ROLES                   AGE    VERSION
tkg-cluster-v1-21-6-control-plane-9fn5c                    Ready    control-plane,master  3d1h   v1.21.6+vmware.1
tkg-cluster-v1-21-6-worker-pool-1-x2szc-7ffbddd567-8ct8m  Ready    <none>                 3d1h   v1.21.6+vmware.1
tkg-cluster-v1-21-6-worker-pool-1-x2szc-7ffbddd567-pn7sp  Ready    <none>                 3d1h   v1.21.6+vmware.1


% kubectl get ns
NAME                             STATUS  AGE
cert-manager                     Active  3d21h
default                          Active  3d22h
kube-node-lease                  Active  3d22h
kube-public                      Active  3d22h
kube-system                      Active  3d22h
tanzu-package-repo-global        Active  3d21h
tanzu-system-dashboard           Active  3d20h
tanzu-system-ingress             Active  3d21h
tanzu-system-monitoring          Active  3d21h
tanzu-system-service-discovery   Active  3d21h
tkg-system                       Active  3d21h
vmware-system-auth               Active  3d22h
vmware-system-cloud-provider     Active  3d22h
vmware-system-csi                Active  3d22h


% kubectl apply -f image-pull-secret.yaml
secret/harbor-registry-secret created


% kubectl get secret
NAME                                  TYPE                                  DATA   AGE
alertmanager-overlay                  Opaque                                1      3d19h
cert-manager-default-sa-token-z7c7q   kubernetes.io/service-account-token   3      3d21h
contour-default-sa-token-d5twq        kubernetes.io/service-account-token   3      3d21h
contour-default-values                Opaque                                1      3d21h
default-token-rkmjd                   kubernetes.io/service-account-token   3      3d22h
external-dns-default-sa-token-78f7j   kubernetes.io/service-account-token   3      3d21h
external-dns-default-values           Opaque                                1      3d21h
grafana-default-sa-token-8t499        kubernetes.io/service-account-token   3      3d20h
grafana-default-values                Opaque                                1      3d20h
harbor-registry-secret                kubernetes.io/dockerconfigjson        1      13s
prometheus-default-sa-token-5srk8     kubernetes.io/service-account-token   3      3d21h
prometheus-default-values             Opaque                                1      3d21h

Step 4. Retrieve images from embedded Harbor in TKG

The Harbor Image Registry secret in the TKG cluster has been created It should now be possible to pull images from the embedded Harbor Image Registry. Let’s try that with a simple busybox pod. Note that the image is being pulled from the embedded Harbor Image Registry. The imagePullSecret references our newly created secret which contains the relevant information to successfully connect to the registry.

% cat busybox-pod-HRBR.yaml
apiVersion: v1
kind: Pod
metadata:
  name: busybox1
  namespace: default
  labels:
    app: busybox1
spec:
  containers:
  - name: busybox
    image: xx.xx.xx.134/cormac-ns/busybox:latest
    command:
      - sleep
      - "3600"
    imagePullPolicy: IfNotPresent
  imagePullSecrets:
  - name: harbor-registry-secret
  restartPolicy: Always


% kubectl apply -f busybox-pod-HRBR.yaml
pod/busybox1 created


% kubectl get pods
NAME      READY  STATUS    RESTARTS  AGE
busybox1  1/1    Running  0          4s

Success! Containers in pods are now able to pull images from the embedded Harbor Image Registry from within the TKG guest/workload cluster.

Note that the embedded Harbor Image Registry has a dependency on NSX-T at the time of writing (vSphere with Tanzu 7.0.3). vSphere with Tanzu with NSX-T networking is available for both on-premises deployments and cloud deployments. The latter is available via Tanzu Services on VMware Cloud on AWS.