Pivotal and Harbor – x509 certificate issues

After deploying and configuring the Harbor tile in Pivotal Ops Manager, I ran into a couple of issues with certificates. The first was encountered when I was  trying to login to harbor from an Ubuntu VM where I was running all of my PKS and BOSH commands. It was also the VM where I pulled my container  images, and the VM from which I now wanted to push them into Harbor. Harbor is our registry server for storing container images. Here is what I got on trying to login:

cormac@pks-cli:~$ sudo docker login -u admin harbor.rainpole.com
Password:

Error response from daemon: Get https://harbor.rainpole.com/v1/users/: \
x509: certificate signed by unknown authority (possibly because of \
"crypto/rsa: verification error" while trying to verify candidate \
authority certificate "Pivotal")

cormac@pks-cli:~$

To resolve this first issue, I had to log into the Harbor UI as the Admin user. From, there I navigated to Administration > Configuration > System Settings, and then I clicked on the Download link associated with the Registry Root Cert, as shown below.

On my Ubuntu VM, the certificate needed to be placed in a particular directory /etc/docker/certs.d/harbor.rainpole.com, where harbor.rainpole.com is obviously the name of my registry that I am trying to login to. With the cert in place, I can now login to my registry, as shown below.

cormac@pks-cli:/etc/docker/certs.d/harbor.rainpole.com$ uname -a
Linux pks-cli.rainpole.com4.13.0-46-generic #51-Ubuntu SMP Tue Jun 12 \
12:36:29 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

cormac@pks-cli:/etc/docker/certs.d/harbor.rainpole.com$ ls
ca.crt

cormac@pks-cli:~$ sudo docker login -u admin harbor.rainpole.com
Password:

Login Succeeded

Cool. At this point, I thought I had solved the certificate issue. I was able to login to Harbor, tag images and push/pull to/from the registry. My next step was to deploy a couchbase app on my Kubernetes cluster, the image of which I had pushed to my registry. However, I got the following issue during the application creation:

root@pks-cli:~/cns-demo# kubectl get pods
NAME          READY   STATUS         RESTARTS   AGE
couchbase-0   0/1     ErrImagePull   0          12s

root@pks-cli:~/cns-demo# kubectl describe pods
Name:               couchbase-0
Namespace:          default
Priority:           0

...

Events:
  Type     Reason                  Age        From                                           \
Message
  ----     ------                  ----       ----                                           \
-------
  Normal   Scheduled               3s         default-scheduler                              \
Successfully assigned default/couchbase-0 to 2e2478da-5a3f-4941-90b1-9410f2cebab2
  Normal   SuccessfulAttachVolume  2s         attachdetach-controller                        \
AttachVolume.Attach succeeded for volume "pvc-b5eb9ff9-2f2b-11e9-805e-00505682e96b"
  Normal   Pulling                 <invalid>  kubelet, 2e2478da-5a3f-4941-90b1-9410f2cebab2  \
pulling image"harbor.rainpole.com/library/saturnism/couchbase:k8s-petset"
  Warning  Failed                  <invalid>  kubelet, 2e2478da-5a3f-4941-90b1-9410f2cebab2  \
Failed to pull image "harbor.rainpole.com/library/saturnism/couchbase:k8s-petset": rpc error:\
 code = Unknown desc = Error response from daemon: Get https://harbor.rainpole.com/v2/: x509:\
 certificate signed by unknown authority
  Warning  Failed                  <invalid>  kubelet, 2e2478da-5a3f-4941-90b1-9410f2cebab2  \
Error: ErrImagePull
  Normal   BackOff                 <invalid>  kubelet, 2e2478da-5a3f-4941-90b1-9410f2cebab2  \
Back-off pulling image "harbor.rainpole.com/library/saturnism/couchbase:k8s-petset"
  Warning  Failed                  <invalid>  kubelet, 2e2478da-5a3f-4941-90b1-9410f2cebab2  \
Error: ImagePullBackOff

root@pks-cli:~/cns-demo#

After some investigation, I found that I missed a step of integrating Harbor with PKS. In a nutshell, I should have copied the contents of my Harbor Registry CA certificate (same certificate I downloaded to my VM) and add it to the BOSH’s list of Trusted Certificates under Security in the BOSH tile in Pivotal Ops Manager. Once I had added it and applied the changes, I was successfully able to deploy my application.

root@pks-cli:~/cns-demo# kubectl get pods
NAME          READY   STATUS    RESTARTS   AGE
couchbase-0   1/1     Running   0          50s

root@pks-cli:~/cns-demo# kubectl describe pods
Name:               couchbase-0
Namespace:          default
Priority:           
...
Events:
  Type     Reason                  Age                From                                           \
Message
  ----     ------                  ----               ----                                           \
-------
  Warning  FailedScheduling        30s (x6 over 37s)  default-scheduler                              \
pod has unbound immediate PersistentVolumeClaims (repeated 3 times)
  Normal   Scheduled               30s                default-scheduler                              \
Successfully assigned default/couchbase-0 to e47914d4-efa3-4087-87f1-f7feb665b324
  Normal   SuccessfulAttachVolume  28s                attachdetach-controller                        \
AttachVolume.Attach succeeded for volume "pvc-8f84d30d-2f8b-11e9-a131-005056821e38"
  Normal   Pulling                 20s                kubelet, e47914d4-efa3-4087-87f1-f7feb665b324  \
pulling image "harbor.rainpole.com/library/saturnism/couchbase:k8s-petset"
  Normal   Pulled                  7s                 kubelet, e47914d4-efa3-4087-87f1-f7feb665b324  \
Successfully pulled image "harbor.rainpole.com/library/saturnism/couchbase:k8s-petset"
  Normal   Created                 7s                 kubelet, e47914d4-efa3-4087-87f1-f7feb665b324  \
Created container
  Normal   Started                 7s                 kubelet, e47914d4-efa3-4087-87f1-f7feb665b324  \
Started container

root@pks-cli:~/cns-demo#