At this point, you may be asking “what is this TKGm thing you are talking about?” and “how is if different to other TKG offerings?” I’m using TKGm to distinguish this as the stand-alone distribution of TKG. This distribution deploys a TKG management cluster, which in turn is used to deploy TKG “workload” clusters. This is different to TKGs, the TKG service which ships with vSphere with Tanzu and VCF with Tanzu. TKGs uses the Supervisor (SV) cluster deployed on vSphere to deploy TKG “guest” or “workload” clusters. Hope that explains it well enough. If you want to learn a bit more about TKGm, and how it all works, check out this earlier post I did on the topic.
Prerequisites
For a successful Harbor deployment, you will need a few items installed on your environment. I am using a VM running Ubuntu 18.04.5. I also have docker and docker-compose installed (docker-compose is used by Harbor). Note that this virtual machine will require internet connectivity to initially pull down a number if container images used for the Harbor deployment. Once the Harbor image registry is deployed, and has been updated to contain all the necessary images for TKGm, then it can be disconnected from the internet. TKG deployment is not covered in this post – I’ll get to it in a future post.
Here are the versions of docker and docker-compose that I am using.
$ docker version Client: Docker Engine - Community Version: 19.03.13 API version: 1.40 Go version: go1.13.15 Git commit: 4484c46d9d Built: Wed Sep 16 17:02:36 2020 OS/Arch: linux/amd64 Experimental: false Server: Docker Engine - Community Engine: Version: 19.03.13 API version: 1.40 (minimum version 1.12) Go version: go1.13.15 Git commit: 4484c46d9d Built: Wed Sep 16 17:01:06 2020 OS/Arch: linux/amd64 Experimental: false containerd: Version: 1.3.7 GitCommit: 8fba4e9a7d01810a393d5d25a3621dc101981175 runc: Version: 1.0.0-rc10 GitCommit: dc9208a3303feef5b3839f4323d9beb36df0a9dd docker-init: Version: 0.18.0 GitCommit: fec3683 $ docker-compose version docker-compose version 1.27.4, build 40524192 docker-py version: 4.3.1 CPython version: 3.7.7 OpenSSL version: OpenSSL 1.1.0l 10 Sep 2019
Certificates and Keys
This is probably the most time consuming part of the whole deployment. To speed things up, I created a few shell scripts. You can find them on GitHub here. The manual procedure, which I am not documenting, can be found in the official Harbor docs here. Setting up the certificates will allow you to login to the Harbor registry securely with https.
Here is the output of the script which prompts for the fully qualified domain name of the host. It creates all the necessary certificates and keys, including your own root CA, to setup Harbor. I have added a few pauses to the script which require the user to hit the enter key to proceed – this is so you can see what is happening at the various points.
$ ./gen-harbor-certs.sh ************************************************************* *** *** *** Script to create CA cert and private key, server CSR, *** *** x509 ext file, and certs for Harbor and Docker *** *** *** ************************************************************* -- Step 0: Checking dependencies ... Type in the fully qualified domain name of the harbor registry (e.g. harbor.vmware.com): cormac-mgmt.corinternal.com Step 1 - Generate a CA Cert Step 1.1 - Generate a CA Cert Private Key Generating RSA private key, 4096 bit long modulus (2 primes) .................++++ .............................++++ e is 65537 (0x010001) Step 1.2 - Generate a CA Cert Certificate Hit enter to continue Step 2 - Generate a Server Certificate Step 2.1 - Generate a Server Certificate Private Key Generating RSA private key, 4096 bit long modulus (2 primes) ..................................................................................................++++ ..++++ e is 65537 (0x010001) Step 2.2 - Generate a Server Certificate Signing Request Step 2.3 - Generate a x509 v3 extension file Step 2.4 - Use the x509 v3 extension file to gerneate a cert for the Harbor host Signature ok subject=C = IE, ST = CORK, L = CORK, O = OCTO, OU = Personal, CN = cormac-mgmt.corinternal.com Getting CA Private Key Hit enter to continue Step 3 - Provide the certificates to Harbor and Docker Step 3.1 - Copy server cert and key to harbor host folder - /data/cert Step 3.2 - Convert .crt to .cert as required by Docker Step 3.3 - Copy server cert and key to docker host folder - /etc/docker/certs.d/cormac-mgmt.corinternal.com Step 4 - Restart Docker Hit enter to continue
Download and Deploy Harbor
Now that we have the certificates in place, we can download, extract and deploy Harbor. I went with Harbor version 2.1.0. Once the Harbor .tgz bundle is downloaded, simply extract it, and make a copy of the Harbor template file (harbor.yml.tmpl) called harbor.yml. Modify this file with details specific to your environment, such as hostname and certificate folder. Here is a view of the files involved.
$ ls harbor-offline-installer-v2.1.0.tgz $ tar zxvf harbor-offline-installer-v2.1.0.tgz harbor/harbor.v2.1.0.tar.gz harbor/prepare harbor/LICENSE harbor/install.sh harbor/common.sh harbor/harbor.yml.tmpl $ cd harbor/ $ ls common.sh harbor.v2.1.0.tar.gz harbor.yml.tmpl install.sh LICENSE prepare $ cp harbor.yml.tmpl harbor.yml
I made the following 3 simple changes to my harbor.yml. I edited the hostname and updated the location of the certificate and key. These were set to /data/cert in the gen-harbor-certs.sh script that we ran earlier. If you modify the script, or choose to place the Harbor cert and key in a different location, you will also need to change the harbor.yml configuration accordingly to reflect this change.
The username and password for the Harbor registry as also configured in this file. I have left them at the default of admin/Harbor12345! You of course, may wish to change these from the default.
# Configuration file of Harbor # The IP address or hostname to access admin UI and registry service. # DO NOT use localhost or 127.0.0.1, because Harbor needs to be accessed by external clients. hostname: cormac-mgmt.corinternal.com # http related config http: # port for http, default is 80. If https enabled, this port will redirect to https port port: 80 # https related config https: # https port for harbor, default is 443 port: 443 # The path of cert and key files for nginx certificate: /data/cert/cormac-mgmt.corinternal.com.crt private_key: /data/cert/cormac-mgmt.corinternal.com.key
Run the Harbor prepare script
Once the harbor.yml configuration is compete, the next step is to run the prepare script included in the Harbor deployment. Note that this pulls down a container to run the task. Since the recent changes to pull limits by Docker, this can be problematic and fail with “too many requests”. A work-around is to be logged in to docker hub with your own docker credentials before doing the pull/prepare. if you do not have any, you can easily create them. Here are the steps:
$ ls
common.sh harbor.v2.1.0.tar.gz harbor.yml harbor.yml.tmpl install.sh LICENSE prepare
$ sudo ./prepare
prepare base dir is set to /home/cormac/harbor/harbor
Unable to find image 'goharbor/prepare:v2.1.0' locally
docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. \
You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.
See 'docker run --help'.
$ docker login
Login with your Docker ID to push and pull images from Docker Hub. \
If you don't have a Docker ID, head over to https://hub.docker.com to create one.
Username:cormachogan
Password:************
WARNING! Your password will be stored unencrypted in /home/cormac/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store
Login Succeeded
$ sudo ./prepare
prepare base dir is set to /home/cormac/harbor/harbor
Generated configuration file: /config/portal/nginx.conf
Generated configuration file: /config/log/logrotate.conf
Generated configuration file: /config/log/rsyslog_docker.conf
Generated configuration file: /config/nginx/nginx.conf
Generated configuration file: /config/core/env
Generated configuration file: /config/core/app.conf
Generated configuration file: /config/registry/config.yml
Generated configuration file: /config/registryctl/env
Generated configuration file: /config/registryctl/config.yml
Generated configuration file: /config/db/env
Generated configuration file: /config/jobservice/env
Generated configuration file: /config/jobservice/config.yml
loaded secret from file: /data/secret/keys/secretkey
Generated configuration file: /compose_location/docker-compose.yml
Clean up the input dir
Install Harbor
The prepare script creates a docker-compose.yml file, which is why we needed to have docker-compose (and docker) installed as part of the prerequisites. We can now run the install.sh script which is also included in the deployment bundle.
cormac@cormac-mgmt:~/harbor/harbor$ ls common common.sh docker-compose.yml harbor.v2.1.0.tar.gz harbor.yml harbor.yml.tmpl install.sh LICENSE prepare cormac@cormac-mgmt:~/harbor/harbor$ sudo ./install.sh [Step 0]: checking if docker is installed ... Note: docker version: 19.03.13 [Step 1]: checking docker-compose is installed ... Note: docker-compose version: 1.27.4 [Step 2]: loading Harbor images ... 643af1ea9877: Loading layer [==================================================>] 4.78MB/4.78MB 2f2eb198019d: Loading layer [==================================================>] 4.096kB/4.096kB 013d878b1cd3: Loading layer [==================================================>] 3.072kB/3.072kB 0cea77165dab: Loading layer [==================================================>] 9.427MB/9.427MB d5eb36d8c949: Loading layer [==================================================>] 10.25MB/10.25MB Loaded image: goharbor/clair-adapter-photon:v2.1.0 . --<snip> . c22a4e901e13: Loading layer [==================================================>] 7.91MB/7.91MB 8214586a2eae: Loading layer [==================================================>] 3.584kB/3.584kB f698b27e2016: Loading layer [==================================================>] 2.56kB/2.56kB 3a34732979ca: Loading layer [==================================================>] 53.04MB/53.04MB 9e8bf761483a: Loading layer [==================================================>] 5.632kB/5.632kB 1b300c7f7b2a: Loading layer [==================================================>] 60.42kB/60.42kB cfe2a4718791: Loading layer [==================================================>] 11.78kB/11.78kB bde4e186d586: Loading layer [==================================================>] 53.87MB/53.87MB b63404705663: Loading layer [==================================================>] 2.56kB/2.56kB Loaded image: goharbor/harbor-core:v2.1.0 [Step 3]: preparing environment ... [Step 4]: preparing harbor configs ... prepare base dir is set to /home/cormac/harbor/harbor Clearing the configuration file: /config/jobservice/config.yml Clearing the configuration file: /config/jobservice/env Clearing the configuration file: /config/portal/nginx.conf Clearing the configuration file: /config/registry/passwd Clearing the configuration file: /config/registry/config.yml Clearing the configuration file: /config/registryctl/config.yml Clearing the configuration file: /config/registryctl/env Clearing the configuration file: /config/db/env Clearing the configuration file: /config/core/app.conf Clearing the configuration file: /config/core/env Clearing the configuration file: /config/nginx/nginx.conf Clearing the configuration file: /config/log/rsyslog_docker.conf Clearing the configuration file: /config/log/logrotate.conf Generated configuration file: /config/portal/nginx.conf Generated configuration file: /config/log/logrotate.conf Generated configuration file: /config/log/rsyslog_docker.conf Generated configuration file: /config/nginx/nginx.conf Generated configuration file: /config/core/env Generated configuration file: /config/core/app.conf Generated configuration file: /config/registry/config.yml Generated configuration file: /config/registryctl/env Generated configuration file: /config/registryctl/config.yml Generated configuration file: /config/db/env Generated configuration file: /config/jobservice/env Generated configuration file: /config/jobservice/config.yml loaded secret from file: /data/secret/keys/secretkey Generated configuration file: /compose_location/docker-compose.yml Clean up the input dir [Step 5]: starting Harbor ... Creating network "harbor_harbor" with the default driver Creating harbor-log ... done Creating redis ... done Creating registry ... done Creating registryctl ... done Creating harbor-portal ... done Creating harbor-db ... done Creating harbor-core ... done Creating harbor-jobservice ... done Creating nginx ... done ✔ ----Harbor has been installed and started successfully.----
This looks good. Note that the commands docker-compose down and docker-compose up can be used to stop and start Harbor if further changes are needed to the configuration files.
Check status of Harbor containers
We can use the docker ps -a command to examine the status of the Harbor containers. In my case, it usually takes ~ 30 seconds for them all to reach a healthy state.
cormac@cormac-mgmt:~/harbor/harbor$ sudo docker ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES b1da486c7e16 goharbor/nginx-photon:v2.1.0 "nginx -g 'daemon of…" 17 seconds ago Up 14 seconds (health: starting) 0.0.0.0:80->8080/tcp, 0.0.0.0:443->8443/tcp nginx ff6d0b00ed66 goharbor/harbor-jobservice:v2.1.0 "/harbor/entrypoint.…" 17 seconds ago Up 15 seconds (health: starting) harbor-jobservice 1d2522c10110 goharbor/harbor-core:v2.1.0 "/harbor/entrypoint.…" 18 seconds ago Up 16 seconds (health: starting) harbor-core 32e2e1246de3 goharbor/harbor-db:v2.1.0 "/docker-entrypoint.…" 22 seconds ago Up 19 seconds (health: starting) harbor-db 11960021374d goharbor/redis-photon:v2.1.0 "redis-server /etc/r…" 22 seconds ago Up 17 seconds (health: starting) redis b14fd57ed92b goharbor/harbor-portal:v2.1.0 "nginx -g 'daemon of…" 22 seconds ago Up 18 seconds (health: starting) harbor-portal 89f947fe7077 goharbor/harbor-registryctl:v2.1.0 "/home/harbor/start.…" 22 seconds ago Up 20 seconds (health: starting) registryctl 8d44c29fc175 goharbor/registry-photon:v2.1.0 "/home/harbor/entryp…" 22 seconds ago Up 19 seconds (health: starting) registry 873df21d0338 goharbor/harbor-log:v2.1.0 "/bin/sh -c /usr/loc…" 23 seconds ago Up 21 seconds (health: starting) 127.0.0.1:1514->10514/tcp harbor-log cormac@cormac-mgmt:~/harbor/harbor$ sudo docker ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES b1da486c7e16 goharbor/nginx-photon:v2.1.0 "nginx -g 'daemon of…" About a minute ago Up About a minute (healthy) 0.0.0.0:80->8080/tcp, 0.0.0.0:443->8443/tcp nginx ff6d0b00ed66 goharbor/harbor-jobservice:v2.1.0 "/harbor/entrypoint.…" About a minute ago Up About a minute (healthy) harbor-jobservice 1d2522c10110 goharbor/harbor-core:v2.1.0 "/harbor/entrypoint.…" About a minute ago Up About a minute (healthy) harbor-core 32e2e1246de3 goharbor/harbor-db:v2.1.0 "/docker-entrypoint.…" About a minute ago Up About a minute (healthy) harbor-db 11960021374d goharbor/redis-photon:v2.1.0 "redis-server /etc/r…" About a minute ago Up About a minute (healthy) redis b14fd57ed92b goharbor/harbor-portal:v2.1.0 "nginx -g 'daemon of…" About a minute ago Up About a minute (healthy) harbor-portal 89f947fe7077 goharbor/harbor-registryctl:v2.1.0 "/home/harbor/start.…" About a minute ago Up About a minute (healthy) registryctl 8d44c29fc175 goharbor/registry-photon:v2.1.0 "/home/harbor/entryp…" About a minute ago Up About a minute (healthy) registry 873df21d0338 goharbor/harbor-log:v2.1.0 "/bin/sh -c /usr/loc…" About a minute ago Up About a minute (healthy) 127.0.0.1:1514->10514/tcp harbor-log
Login to the Harbor registry
Next, let’s login to the Harbor registry. I will verify it by using both http and https.
cormac@cormac-mgmt:~$ docker login http://cormac-mgmt.corinternal.com Username: admin Password: ***** WARNING! Your password will be stored unencrypted in /home/cormac/.docker/config.json. Configure a credential helper to remove this warning. See https://docs.docker.com/engine/reference/commandline/login/#credentials-store Login Succeeded cormac@cormac-mgmt:~$ docker login https://cormac-mgmt.corinternal.com Authenticating with existing credentials... WARNING! Your password will be stored unencrypted in /home/cormac/.docker/config.json. Configure a credential helper to remove this warning. See https://docs.docker.com/engine/reference/commandline/login/#credentials-store Login Succeeded
Note that I have used the fully qualified domain name here. If you wish to use shorter names for the registry, e.g. cormac-mgmt, then I will need to add these additional DNS entries to the X509 v3.ext file which was configured when we ran the certificate script. See the Harbor docs for details on how to do this.
Push a container image to Harbor
In this example, I am going to run the simple docker hello-world container. I will then tag it, and push it up to the Harbor registry to verify that everything is working as expected. Note that I experienced the same issue as before when trying to run the hello-world container – docker complaining that I have reached my pull rate limit. I once again login to docker hub with my docker credentials. Now I can run the container by pulling the image from docker.
cormac@cormac-mgmt:~$ docker run hello-world Unable to find image 'hello-world:latest' locally docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. \ You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. See 'docker run --help'. cormac@cormac-mgmt:~$ docker login Login with your Docker ID to push and pull images from Docker Hub. \ If you don't have a Docker ID, head over to https://hub.docker.com to create one. Username: cormachogan Password: ********** WARNING! Your password will be stored unencrypted in /home/cormac/.docker/config.json. Configure a credential helper to remove this warning. See https://docs.docker.com/engine/reference/commandline/login/#credentials-store Login Succeeded cormac@cormac-mgmt:~$ docker run hello-world Unable to find image 'hello-world:latest' locally latest: Pulling from library/hello-world 0e03bdcc26d7: Pull complete Digest: sha256:e7c70bb24b462baa86c102610182e3efcb12a04854e8c582838d92970a09f323 Status: Downloaded newer image for hello-world:latest Hello from Docker! This message shows that your installation appears to be working correctly. To generate this message, Docker took the following steps: 1. The Docker client contacted the Docker daemon. 2. The Docker daemon pulled the "hello-world" image from the Docker Hub. (amd64) 3. The Docker daemon created a new container from that image which runs the executable that produces the output you are currently reading. 4. The Docker daemon streamed that output to the Docker client, which sent it to your terminal. To try something more ambitious, you can run an Ubuntu container with: $ docker run -it ubuntu bash Share images, automate workflows, and more with a free Docker ID: https://hub.docker.com/ For more examples and ideas, visit: https://docs.docker.com/get-started/ cormac@cormac-mgmt:~$
Let’s now tag this container image and push it up to my Harbor image registry. I am showing another login, but this may not be necessary if you are already logged into the registry from an earlier step.
cormac@cormac-mgmt:~$ sudo docker tag hello-world:latest cormac-mgmt.corinternal.com/library/hello-world:latest cormac@cormac-mgmt:~$ sudo docker login https://cormac-mgmt.corinternal.com Username: admin Password: ****** WARNING! Your password will be stored unencrypted in /home/cormac/.docker/config.json. Configure a credential helper to remove this warning. See https://docs.docker.com/engine/reference/commandline/login/#credentials-store Login Succeeded cormac@cormac-mgmt:~$ sudo docker push cormac-mgmt.corinternal.com/library/hello-world:latest The push refers to repository [cormac-mgmt.corinternal.com/library/hello-world] 9c27e219663c: Pushed latest: digest: sha256:90659bf80b44ce6be8234e6ff90a1ac34acbeb826903b02cfa0da11c82cbc042 size: 525
Everything looks successful. One other check would be to remove the image locally and attempt to run it once again using the image in the Harbor registry. This will pull the image from the Harbor registry. Let’s try that now.
$ docker images | grep hello-world cormac-mgmt.corinternal.com/library/hello-world latest bf756fb1ae65 11 months ago 13.3kB hello-world latest bf756fb1ae65 11 months ago 13.3kB $ docker rmi hello-world:latest Untagged: hello-world:latest Untagged: hello-world@sha256:e7c70bb24b462baa86c102610182e3efcb12a04854e8c582838d92970a09f323 $ docker rmi cormac-mgmt.corinternal.com/library/hello-world:latest Untagged: cormac-mgmt.corinternal.com/library/hello-world:latest Untagged: cormac-mgmt.corinternal.com/library/hello-world@sha256:90659bf80b44ce6be8234e6ff90a1ac34acbeb826903b02cfa0da11c82cbc042 Deleted: sha256:bf756fb1ae65adf866bd8c456593cd24beb6a0a061dedf42b26a993176745f6b Deleted: sha256:9c27e219663c25e0f28493790cc0b88bc973ba3b1686355f221c38a36978ac63 $ docker images | grep hello-world $ $ docker run cormac-mgmt.corinternal.com/library/hello-world:latest Unable to find image 'cormac-mgmt.corinternal.com/library/hello-world:latest' locally latest: Pulling from library/hello-world 0e03bdcc26d7: Pull complete Digest: sha256:90659bf80b44ce6be8234e6ff90a1ac34acbeb826903b02cfa0da11c82cbc042 Status: Downloaded newer image for cormac-mgmt.corinternal.com/library/hello-world:latest Hello from Docker! This message shows that your installation appears to be working correctly. To generate this message, Docker took the following steps: 1. The Docker client contacted the Docker daemon. 2. The Docker daemon pulled the "hello-world" image from the Docker Hub. (amd64) 3. The Docker daemon created a new container from that image which runs the executable that produces the output you are currently reading. 4. The Docker daemon streamed that output to the Docker client, which sent it to your terminal. To try something more ambitious, you can run an Ubuntu container with: $ docker run -it ubuntu bash Share images, automate workflows, and more with a free Docker ID: https://hub.docker.com/ For more examples and ideas, visit: https://docs.docker.com/get-started/ $
Looks good. We can do one last check and use the UI to verify that the image was pushed up to the library project on our Harbor registry. Simply point a browser at the URL we have been using for the registry, in my case https://cormac-tkgm.corinternal.com, and log in and check the contents of the library project for any images.
Everything looks good. Harbor is deployed locally, we can login to it using https, and we can push images to it. Everything is in place to start the remaining steps to setup an air-gapped, internet restricted TKGm deployment. I will cover this in a future blog post very shortly.
[Update] If you want to control the bring up Harbor on reboot, check out this blog post by Chip Zoller which shows how to create systemd unit file to run a docker-compose on startup and shutdown to control the stopping and starting of Harbor.