Site icon CormacHogan.com

Deploying Harbor v2.1.0 – Step By Step

Over the thanksgiving break, I took the opportunity to look at the steps required to deploying Tanzu Kubernetes Grid (TKGm) in an air-gapped or internet-restricted environment. The first step to achieving this was to deploy the Harbor Container Image Registry locally in my own environment. While I’ve written about Harbor quite a bit in the early days, I haven’t looked at it in earnest recently, so it was good to revisit it and see what changed. In this post, I’ll walk through the steps involved, and point you to few scripts that I developed to speed up the process.

At this point, you may be asking “what is this TKGm thing you are talking about?” and “how is if different to other TKG offerings?” I’m using TKGm to distinguish this as the stand-alone distribution of TKG. This distribution deploys a TKG management cluster, which in turn is used to deploy TKG “workload” clusters. This is different to TKGs, the TKG service which ships with vSphere with Tanzu and VCF with Tanzu. TKGs uses the Supervisor (SV) cluster deployed on vSphere to deploy TKG “guest” or “workload” clusters. Hope that explains it well enough. If you want to learn a bit more about TKGm, and how it all works, check out this earlier post I did on the topic.

Prerequisites

For a successful Harbor deployment, you will need a few items installed on your environment. I am using a  VM running Ubuntu 18.04.5. I also have docker and docker-compose installed (docker-compose is used by Harbor). Note that this virtual machine will require internet connectivity to initially pull down a number if container images used for the Harbor deployment. Once the Harbor image registry is deployed, and has been updated to contain all the necessary images for TKGm, then it can be disconnected from the internet. TKG deployment is not covered in this post – I’ll get to it in a future post.

Here are the versions of docker and docker-compose that I am using.

$ docker version
Client: Docker Engine - Community
Version:           19.03.13
API version:       1.40
Go version:        go1.13.15
Git commit:        4484c46d9d
Built:             Wed Sep 16 17:02:36 2020
OS/Arch:           linux/amd64
Experimental:      false

Server: Docker Engine - Community
Engine:
  Version:          19.03.13
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.13.15
  Git commit:       4484c46d9d
  Built:            Wed Sep 16 17:01:06 2020
  OS/Arch:          linux/amd64
  Experimental:     false
containerd:
  Version:          1.3.7
  GitCommit:        8fba4e9a7d01810a393d5d25a3621dc101981175
runc:
  Version:          1.0.0-rc10
  GitCommit:        dc9208a3303feef5b3839f4323d9beb36df0a9dd
docker-init:
  Version:          0.18.0
  GitCommit:        fec3683


$ docker-compose version
docker-compose version 1.27.4, build 40524192
docker-py version: 4.3.1
CPython version: 3.7.7
OpenSSL version: OpenSSL 1.1.0l  10 Sep 2019

Certificates and Keys

This is probably the most time consuming part of the whole deployment. To speed things up, I created a few shell scripts. You can find them on GitHub here. The manual procedure, which I am not documenting, can be found in the official Harbor docs here. Setting up the certificates will allow you to login to the Harbor registry securely with https.

Here is the output of the script which prompts for the fully qualified domain name of the host. It creates all the necessary certificates and keys, including your own root CA, to setup Harbor. I have added a few pauses to the script which require the user to hit the enter key to proceed – this is so you can see what is happening at the various points.

$ ./gen-harbor-certs.sh

*************************************************************
***                                                       ***
*** Script to create CA cert and private key, server CSR, ***
*** x509 ext file, and certs for Harbor and Docker        ***
***                                                       ***
*************************************************************

-- Step 0: Checking dependencies ...

Type in the fully qualified domain name of the harbor registry (e.g. harbor.vmware.com):
cormac-mgmt.corinternal.com
Step 1 - Generate a CA Cert

Step 1.1 -  Generate a CA Cert Private Key

Generating RSA private key, 4096 bit long modulus (2 primes)
.................++++
.............................++++
e is 65537 (0x010001)


Step 1.2 -  Generate a CA Cert Certificate

Hit enter to continue


Step 2 - Generate a Server Certificate

Step 2.1 - Generate a Server Certificate Private Key

Generating RSA private key, 4096 bit long modulus (2 primes)
..................................................................................................++++
..++++
e is 65537 (0x010001)


Step 2.2 - Generate a Server Certificate Signing Request

Step 2.3 - Generate a x509 v3 extension file

Step 2.4 - Use the x509 v3 extension file to gerneate a cert for the Harbor host

Signature ok

subject=C = IE, ST = CORK, L = CORK, O = OCTO, OU = Personal, CN = cormac-mgmt.corinternal.com
Getting CA Private Key

Hit enter to continue


Step 3 - Provide the certificates to Harbor and Docker

Step 3.1 - Copy server cert and key to harbor host folder - /data/cert

Step 3.2 - Convert .crt to .cert as required by Docker

Step 3.3 - Copy server cert and key to docker host folder - /etc/docker/certs.d/cormac-mgmt.corinternal.com

Step 4 -  Restart Docker

Hit enter to continue

Download and Deploy Harbor

Now that we have the certificates in place, we can download, extract and deploy Harbor. I went with Harbor version 2.1.0. Once the Harbor .tgz bundle is downloaded, simply extract it, and make a copy of the Harbor template file (harbor.yml.tmpl) called harbor.yml. Modify this file with details specific to your environment, such as hostname and certificate folder. Here is a view of the files involved.

$ ls
harbor-offline-installer-v2.1.0.tgz

$ tar zxvf harbor-offline-installer-v2.1.0.tgz
harbor/harbor.v2.1.0.tar.gz
harbor/prepare
harbor/LICENSE
harbor/install.sh
harbor/common.sh
harbor/harbor.yml.tmpl

$ cd harbor/

$ ls
common.sh  harbor.v2.1.0.tar.gz  harbor.yml.tmpl  install.sh  LICENSE  prepare

$ cp harbor.yml.tmpl harbor.yml

I made the following 3 simple changes to my harbor.yml. I edited the hostname and updated the location of the certificate and key. These were set to /data/cert in the gen-harbor-certs.sh script that we ran earlier. If you modify the script, or  choose to place the Harbor cert and key in a different location, you will also need to change the harbor.yml configuration accordingly to reflect this change.

The username and password for the Harbor registry as also configured in this file. I have left them at the default of admin/Harbor12345! You of course, may wish to change these from the default.

# Configuration file of Harbor

# The IP address or hostname to access admin UI and registry service.
# DO NOT use localhost or 127.0.0.1, because Harbor needs to be accessed by external clients.
hostname: cormac-mgmt.corinternal.com
# http related config
http:
  # port for http, default is 80. If https enabled, this port will redirect to https port
  port: 80

# https related config
https:
  # https port for harbor, default is 443
  port: 443
  # The path of cert and key files for nginx
  certificate: /data/cert/cormac-mgmt.corinternal.com.crt
  private_key: /data/cert/cormac-mgmt.corinternal.com.key

Run the Harbor prepare script

Once the harbor.yml configuration is compete, the next step is to run the prepare script included in the Harbor deployment. Note that this pulls down a container to run the task. Since the recent changes to pull limits by Docker, this can be problematic and fail with “too many requests”. A work-around is to be logged in to docker hub with your own docker credentials before doing the pull/prepare. if you do not have any, you can easily create them. Here are the steps:

$ ls
common.sh  harbor.v2.1.0.tar.gz  harbor.yml  harbor.yml.tmpl  install.sh  LICENSE  prepare

$ sudo ./prepare
prepare base dir is set to /home/cormac/harbor/harbor
Unable to find image 'goharbor/prepare:v2.1.0' locally
docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. \
You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.
See 'docker run --help'.

$ docker login
Login with your Docker ID to push and pull images from Docker Hub. \
If you don't have a Docker ID, head over to https://hub.docker.com to create one.
Username:cormachogan
Password:************

WARNING! Your password will be stored unencrypted in /home/cormac/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store
Login Succeeded

$ sudo ./prepare
prepare base dir is set to /home/cormac/harbor/harbor
Generated configuration file: /config/portal/nginx.conf
Generated configuration file: /config/log/logrotate.conf
Generated configuration file: /config/log/rsyslog_docker.conf
Generated configuration file: /config/nginx/nginx.conf
Generated configuration file: /config/core/env
Generated configuration file: /config/core/app.conf
Generated configuration file: /config/registry/config.yml
Generated configuration file: /config/registryctl/env
Generated configuration file: /config/registryctl/config.yml
Generated configuration file: /config/db/env
Generated configuration file: /config/jobservice/env
Generated configuration file: /config/jobservice/config.yml
loaded secret from file: /data/secret/keys/secretkey
Generated configuration file: /compose_location/docker-compose.yml
Clean up the input dir 

Install Harbor

The prepare script creates a docker-compose.yml file, which is why we needed to have docker-compose (and docker) installed as part of the prerequisites. We can now run the install.sh script which is also included in the deployment bundle.

cormac@cormac-mgmt:~/harbor/harbor$ ls
common  common.sh  docker-compose.yml  harbor.v2.1.0.tar.gz  harbor.yml  harbor.yml.tmpl  install.sh  LICENSE  prepare

cormac@cormac-mgmt:~/harbor/harbor$ sudo ./install.sh
[Step 0]: checking if docker is installed ...
Note: docker version: 19.03.13

[Step 1]: checking docker-compose is installed ...
Note: docker-compose version: 1.27.4

[Step 2]: loading Harbor images ...
643af1ea9877: Loading layer [==================================================>]   4.78MB/4.78MB
2f2eb198019d: Loading layer [==================================================>]  4.096kB/4.096kB
013d878b1cd3: Loading layer [==================================================>]  3.072kB/3.072kB
0cea77165dab: Loading layer [==================================================>]  9.427MB/9.427MB
d5eb36d8c949: Loading layer [==================================================>]  10.25MB/10.25MB
Loaded image: goharbor/clair-adapter-photon:v2.1.0
.
--<snip>
.
c22a4e901e13: Loading layer [==================================================>]   7.91MB/7.91MB
8214586a2eae: Loading layer [==================================================>]  3.584kB/3.584kB
f698b27e2016: Loading layer [==================================================>]   2.56kB/2.56kB
3a34732979ca: Loading layer [==================================================>]  53.04MB/53.04MB
9e8bf761483a: Loading layer [==================================================>]  5.632kB/5.632kB
1b300c7f7b2a: Loading layer [==================================================>]  60.42kB/60.42kB
cfe2a4718791: Loading layer [==================================================>]  11.78kB/11.78kB
bde4e186d586: Loading layer [==================================================>]  53.87MB/53.87MB
b63404705663: Loading layer [==================================================>]   2.56kB/2.56kB
Loaded image: goharbor/harbor-core:v2.1.0

[Step 3]: preparing environment ...

[Step 4]: preparing harbor configs ...
prepare base dir is set to /home/cormac/harbor/harbor
Clearing the configuration file: /config/jobservice/config.yml
Clearing the configuration file: /config/jobservice/env
Clearing the configuration file: /config/portal/nginx.conf
Clearing the configuration file: /config/registry/passwd
Clearing the configuration file: /config/registry/config.yml
Clearing the configuration file: /config/registryctl/config.yml
Clearing the configuration file: /config/registryctl/env
Clearing the configuration file: /config/db/env
Clearing the configuration file: /config/core/app.conf
Clearing the configuration file: /config/core/env
Clearing the configuration file: /config/nginx/nginx.conf
Clearing the configuration file: /config/log/rsyslog_docker.conf
Clearing the configuration file: /config/log/logrotate.conf
Generated configuration file: /config/portal/nginx.conf
Generated configuration file: /config/log/logrotate.conf
Generated configuration file: /config/log/rsyslog_docker.conf
Generated configuration file: /config/nginx/nginx.conf
Generated configuration file: /config/core/env
Generated configuration file: /config/core/app.conf
Generated configuration file: /config/registry/config.yml
Generated configuration file: /config/registryctl/env
Generated configuration file: /config/registryctl/config.yml
Generated configuration file: /config/db/env
Generated configuration file: /config/jobservice/env
Generated configuration file: /config/jobservice/config.yml
loaded secret from file: /data/secret/keys/secretkey
Generated configuration file: /compose_location/docker-compose.yml
Clean up the input dir

[Step 5]: starting Harbor ...
Creating network "harbor_harbor" with the default driver
Creating harbor-log ... done
Creating redis         ... done
Creating registry      ... done
Creating registryctl   ... done
Creating harbor-portal ... done
Creating harbor-db     ... done
Creating harbor-core   ... done
Creating harbor-jobservice ... done
Creating nginx             ... done

✔ ----Harbor has been installed and started successfully.----

This looks good. Note that the commands docker-compose down and docker-compose up can be used to stop and start Harbor if further changes are needed to the configuration files.

Check status of Harbor containers

We can use the docker ps -a command to examine the status of the Harbor containers. In my case, it usually takes ~ 30 seconds for them all to reach a healthy state.

cormac@cormac-mgmt:~/harbor/harbor$ sudo docker ps -a
CONTAINER ID        IMAGE                                COMMAND                  CREATED             STATUS                             PORTS                                         NAMES
b1da486c7e16        goharbor/nginx-photon:v2.1.0         "nginx -g 'daemon of…"   17 seconds ago      Up 14 seconds (health: starting)   0.0.0.0:80->8080/tcp, 0.0.0.0:443->8443/tcp   nginx
ff6d0b00ed66        goharbor/harbor-jobservice:v2.1.0    "/harbor/entrypoint.…"   17 seconds ago      Up 15 seconds (health: starting)                                                 harbor-jobservice
1d2522c10110        goharbor/harbor-core:v2.1.0          "/harbor/entrypoint.…"   18 seconds ago      Up 16 seconds (health: starting)                                                 harbor-core
32e2e1246de3        goharbor/harbor-db:v2.1.0            "/docker-entrypoint.…"   22 seconds ago      Up 19 seconds (health: starting)                                                 harbor-db
11960021374d        goharbor/redis-photon:v2.1.0         "redis-server /etc/r…"   22 seconds ago      Up 17 seconds (health: starting)                                                 redis
b14fd57ed92b        goharbor/harbor-portal:v2.1.0        "nginx -g 'daemon of…"   22 seconds ago      Up 18 seconds (health: starting)                                                 harbor-portal
89f947fe7077        goharbor/harbor-registryctl:v2.1.0   "/home/harbor/start.…"   22 seconds ago      Up 20 seconds (health: starting)                                                 registryctl
8d44c29fc175        goharbor/registry-photon:v2.1.0      "/home/harbor/entryp…"   22 seconds ago      Up 19 seconds (health: starting)                                                 registry
873df21d0338        goharbor/harbor-log:v2.1.0           "/bin/sh -c /usr/loc…"   23 seconds ago      Up 21 seconds (health: starting)   127.0.0.1:1514->10514/tcp                     harbor-log

cormac@cormac-mgmt:~/harbor/harbor$ sudo docker ps -a
CONTAINER ID        IMAGE                                COMMAND                  CREATED              STATUS                        PORTS                                         NAMES
b1da486c7e16        goharbor/nginx-photon:v2.1.0         "nginx -g 'daemon of…"   About a minute ago   Up About a minute (healthy)   0.0.0.0:80->8080/tcp, 0.0.0.0:443->8443/tcp   nginx
ff6d0b00ed66        goharbor/harbor-jobservice:v2.1.0    "/harbor/entrypoint.…"   About a minute ago   Up About a minute (healthy)                                                 harbor-jobservice
1d2522c10110        goharbor/harbor-core:v2.1.0          "/harbor/entrypoint.…"   About a minute ago   Up About a minute (healthy)                                                 harbor-core
32e2e1246de3        goharbor/harbor-db:v2.1.0            "/docker-entrypoint.…"   About a minute ago   Up About a minute (healthy)                                                 harbor-db
11960021374d        goharbor/redis-photon:v2.1.0         "redis-server /etc/r…"   About a minute ago   Up About a minute (healthy)                                                 redis
b14fd57ed92b        goharbor/harbor-portal:v2.1.0        "nginx -g 'daemon of…"   About a minute ago   Up About a minute (healthy)                                                 harbor-portal
89f947fe7077        goharbor/harbor-registryctl:v2.1.0   "/home/harbor/start.…"   About a minute ago   Up About a minute (healthy)                                                 registryctl
8d44c29fc175        goharbor/registry-photon:v2.1.0      "/home/harbor/entryp…"   About a minute ago   Up About a minute (healthy)                                                 registry
873df21d0338        goharbor/harbor-log:v2.1.0           "/bin/sh -c /usr/loc…"   About a minute ago   Up About a minute (healthy)   127.0.0.1:1514->10514/tcp                     harbor-log

Login to the Harbor registry

Next, let’s login to the Harbor registry. I will verify it by using both http and https.

cormac@cormac-mgmt:~$ docker login http://cormac-mgmt.corinternal.com
Username: admin
Password: *****
WARNING! Your password will be stored unencrypted in /home/cormac/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store
Login Succeeded

cormac@cormac-mgmt:~$ docker login https://cormac-mgmt.corinternal.com
Authenticating with existing credentials...
WARNING! Your password will be stored unencrypted in /home/cormac/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store
Login Succeeded

Note that I have used the fully qualified domain name here. If you wish to use shorter names for the registry, e.g. cormac-mgmt, then I will need to add these additional DNS entries to the X509 v3.ext file which was configured when we ran the certificate script. See the Harbor docs for details on how to do this.

Push a container image to Harbor

In this example, I am going to run the simple docker hello-world container. I will then tag it, and push it up to the Harbor registry to verify that everything is working as expected. Note that I experienced the same issue as before when trying to run the hello-world container – docker complaining that I have reached my pull rate limit. I once again login to docker hub with my docker credentials. Now I can run the container by pulling the image from docker.

cormac@cormac-mgmt:~$ docker run hello-world
Unable to find image 'hello-world:latest' locally
docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. \
You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.
See 'docker run --help'.


cormac@cormac-mgmt:~$ docker login
Login with your Docker ID to push and pull images from Docker Hub. \
If you don't have a Docker ID, head over to https://hub.docker.com to create one.
Username: cormachogan
Password: **********

WARNING! Your password will be stored unencrypted in /home/cormac/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store

Login Succeeded

cormac@cormac-mgmt:~$ docker run hello-world
Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
0e03bdcc26d7: Pull complete
Digest: sha256:e7c70bb24b462baa86c102610182e3efcb12a04854e8c582838d92970a09f323
Status: Downloaded newer image for hello-world:latest

Hello from Docker!
This message shows that your installation appears to be working correctly.

To generate this message, Docker took the following steps:
1. The Docker client contacted the Docker daemon.
2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
    (amd64)
3. The Docker daemon created a new container from that image which runs the
    executable that produces the output you are currently reading.
4. The Docker daemon streamed that output to the Docker client, which sent it
    to your terminal.

To try something more ambitious, you can run an Ubuntu container with:

$ docker run -it ubuntu bash

Share images, automate workflows, and more with a free Docker ID:
https://hub.docker.com/

For more examples and ideas, visit:
https://docs.docker.com/get-started/

cormac@cormac-mgmt:~$

Let’s now tag this container image and push it up to my Harbor image registry. I am showing another login, but this may not be necessary if you are already logged into the registry from an earlier step.

cormac@cormac-mgmt:~$ sudo docker tag hello-world:latest cormac-mgmt.corinternal.com/library/hello-world:latest

cormac@cormac-mgmt:~$ sudo docker login https://cormac-mgmt.corinternal.com
Username: admin
Password: ******

WARNING! Your password will be stored unencrypted in /home/cormac/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store

Login Succeeded

cormac@cormac-mgmt:~$ sudo docker push cormac-mgmt.corinternal.com/library/hello-world:latest
The push refers to repository [cormac-mgmt.corinternal.com/library/hello-world]
9c27e219663c: Pushed
latest: digest: sha256:90659bf80b44ce6be8234e6ff90a1ac34acbeb826903b02cfa0da11c82cbc042 size: 525

Everything looks successful. One other check would be to remove the image locally and attempt to run it once again using the image in the Harbor registry. This will pull the image from the Harbor registry. Let’s try that now.

$ docker images | grep hello-world
cormac-mgmt.corinternal.com/library/hello-world                                       latest                         bf756fb1ae65        11 months ago       13.3kB
hello-world                                                                           latest                         bf756fb1ae65        11 months ago       13.3kB

$ docker rmi hello-world:latest
Untagged: hello-world:latest
Untagged: hello-world@sha256:e7c70bb24b462baa86c102610182e3efcb12a04854e8c582838d92970a09f323

$ docker rmi cormac-mgmt.corinternal.com/library/hello-world:latest
Untagged: cormac-mgmt.corinternal.com/library/hello-world:latest
Untagged: cormac-mgmt.corinternal.com/library/hello-world@sha256:90659bf80b44ce6be8234e6ff90a1ac34acbeb826903b02cfa0da11c82cbc042
Deleted: sha256:bf756fb1ae65adf866bd8c456593cd24beb6a0a061dedf42b26a993176745f6b
Deleted: sha256:9c27e219663c25e0f28493790cc0b88bc973ba3b1686355f221c38a36978ac63

$ docker images | grep hello-world
$

$ docker run cormac-mgmt.corinternal.com/library/hello-world:latest
Unable to find image 'cormac-mgmt.corinternal.com/library/hello-world:latest' locally
latest: Pulling from library/hello-world
0e03bdcc26d7: Pull complete
Digest: sha256:90659bf80b44ce6be8234e6ff90a1ac34acbeb826903b02cfa0da11c82cbc042
Status: Downloaded newer image for cormac-mgmt.corinternal.com/library/hello-world:latest

Hello from Docker!
This message shows that your installation appears to be working correctly.

To generate this message, Docker took the following steps:
1. The Docker client contacted the Docker daemon.
2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
    (amd64)
3. The Docker daemon created a new container from that image which runs the
    executable that produces the output you are currently reading.
4. The Docker daemon streamed that output to the Docker client, which sent it
    to your terminal.

To try something more ambitious, you can run an Ubuntu container with:
$ docker run -it ubuntu bash

Share images, automate workflows, and more with a free Docker ID:
https://hub.docker.com/

For more examples and ideas, visit:
https://docs.docker.com/get-started/
$

Looks good. We can do one last check and use the UI to verify that the image was pushed up to the library project on our Harbor registry. Simply point a browser at the URL we have been using for the registry, in my case https://cormac-tkgm.corinternal.com, and log in and check the contents of the library project for any images.

Everything looks good. Harbor is deployed locally, we can login to it using https, and we can push images to it. Everything is in place to start the remaining steps to setup an air-gapped, internet restricted TKGm deployment. I will cover this in a future blog post very shortly.

[Update] If you want to control the bring up Harbor on reboot, check out this blog post by Chip Zoller which shows how to create systemd unit file to run a docker-compose on startup and shutdown to control the stopping and starting of Harbor.

Exit mobile version