Pivotal and Harbor – x509 certificate issues

After deploying and configuring the Harbor tile in Pivotal Ops Manager, I ran into a couple of issues with certificates. The first was encountered when I was  trying to login to harbor from an Ubuntu VM where I was running all of my PKS and BOSH commands. It was also the VM where I pulled my container  images, and the VM from which I now wanted to push them into Harbor. Harbor is our registry server for storing container images. Here is what I got on trying to login:

 

cormac@pks-cli:~$ sudo docker login -u admin harbor.rainpole.com
Password:
Error response from daemon: Get https://harbor.rainpole.com/v1/users/: x509: certificate signed by unknown authority (possibly because of “crypto/rsa: verification error” while trying to verify candidate authority certificate “Pivotal”)
cormac@pks-cli:~$

 

To resolve this first issue, I had to log into the Harbor UI as the Admin user. From, there I navigated to Administration > Configuration > System Settings, and then I clicked on the Download link associated with the Registry Root Cert, as shown below.
On my Ubuntu VM, the certificate needed to be placed in a particular directory /etc/docker/certs.d/harbor.rainpole.com, where harbor.rainpole.com is obviously the name of my registry that I am trying to login to. With the cert in place, I can now login to my registry, as shown below.
cormac@pks-cli:/etc/docker/certs.d/harbor.rainpole.com$ uname -a
Linux pks-cli.rainpole.com4.13.0-46-generic #51-Ubuntu SMP Tue Jun 12 12:36:29 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
cormac@pks-cli:/etc/docker/certs.d/harbor.rainpole.com$ ls
ca.crt
cormac@pks-cli:/etc/docker/certs.d/harbor.rainpole.com$ sudo docker login -u admin harbor.rainpole.com
Password:
Login Succeeded

 

Cool. At this point, I thought I had solved the certificate issue. I was able to login to Harbor, tag images and push/pull to/from the registry. My next step was to deploy a couchbase app on my Kubernetes cluster, the image of which I had pushed to my registry. However, I got the following issue during the application creation:
root@pks-cli:~/cns-demo# kubectl get pods
NAME          READY   STATUS         RESTARTS   AGE
couchbase-0   0/1     ErrImagePull   0          12s
root@pks-cli:~/cns-demo# kubectl describe pods
Name:               couchbase-0
Namespace:          default
Priority:           0

.

.
.
Events:
  Type     Reason                  Age        From                                           Message
  —-     ——                  —-       —-                                           ——-
  Normal   Scheduled               3s         default-scheduler                              Successfully assigned default/couchbase-0 to 2e2478da-5a3f-4941-90b1-9410f2cebab2
  Normal   SuccessfulAttachVolume  2s         attachdetach-controller                        AttachVolume.Attach succeeded for volume “pvc-b5eb9ff9-2f2b-11e9-805e-00505682e96b”
  Normal   Pulling                 <invalid>  kubelet, 2e2478da-5a3f-4941-90b1-9410f2cebab2  pulling image”harbor.rainpole.com/library/saturnism/couchbase:k8s-petset”
  Warning  Failed                  <invalid>  kubelet, 2e2478da-5a3f-4941-90b1-9410f2cebab2  Failed to pull image “harbor.rainpole.com/library/saturnism/couchbase:k8s-petset”: rpc error: code = Unknown desc = Error response from daemon: Get https://harbor.rainpole.com/v2/: x509: certificate signed by unknown authority
  Warning  Failed                  <invalid>  kubelet, 2e2478da-5a3f-4941-90b1-9410f2cebab2  Error: ErrImagePull
  Normal   BackOff                 <invalid>  kubelet, 2e2478da-5a3f-4941-90b1-9410f2cebab2  Back-off pulling image “harbor.rainpole.com/library/saturnism/couchbase:k8s-petset”
  Warning  Failed                  <invalid>  kubelet, 2e2478da-5a3f-4941-90b1-9410f2cebab2  Error: ImagePullBackOff
root@pks-cli:~/cns-demo#

 

After some investigation, I found that I missed a step of integrating Harbor with PKS. In a nutshell, I should have copied the contents of my Harbor Registry CA certificate (same certificate I downloaded to my VM) and add it to the BOSH’s list of Trusted Certificates under Security in the BOSH tile in Pivotal Ops Manager. Once I had added it and applied the changes, I was successfully able to deploy my application.

 

root@pks-cli:~/cns-demo# kubectl get pods
NAME          READY   STATUS    RESTARTS   AGE
couchbase-0   1/1     Running   0          50s
root@pks-cli:~/cns-demo# kubectl describe pods
Name:               couchbase-0
Namespace:          default
Priority:           0

.

.
.
Events:
  Type     Reason                  Age                From                                           Message
  —-     ——                  —-               —-                                           ——-
  Warning  FailedScheduling        30s (x6 over 37s)  default-scheduler                              pod has unbound immediate PersistentVolumeClaims (repeated 3 times)
  Normal   Scheduled               30s                default-scheduler                              Successfully assigned default/couchbase-0 to e47914d4-efa3-4087-87f1-f7feb665b324
  Normal   SuccessfulAttachVolume  28s                attachdetach-controller                        AttachVolume.Attach succeeded for volume “pvc-8f84d30d-2f8b-11e9-a131-005056821e38”
  Normal   Pulling                 20s                kubelet, e47914d4-efa3-4087-87f1-f7feb665b324  pulling image “harbor.rainpole.com/library/saturnism/couchbase:k8s-petset”
  Normal   Pulled                  7s                 kubelet, e47914d4-efa3-4087-87f1-f7feb665b324  Successfully pulled image “harbor.rainpole.com/library/saturnism/couchbase:k8s-petset”
  Normal   Created                 7s                 kubelet, e47914d4-efa3-4087-87f1-f7feb665b324  Created container
  Normal   Started                 7s                 kubelet, e47914d4-efa3-4087-87f1-f7feb665b324  Started container
root@pks-cli:~/cns-demo#