Pivotal and Harbor – x509 certificate issues
After deploying and configuring the Harbor tile in Pivotal Ops Manager, I ran into a couple of issues with certificates. The first was encountered when I was trying to login to harbor from an Ubuntu VM where I was running all of my PKS and BOSH commands. It was also the VM where I pulled my container images, and the VM from which I now wanted to push them into Harbor. Harbor is our registry server for storing container images. Here is what I got on trying to login:
cormac@pks-cli:~$ sudo docker login -u admin harbor.rainpole.com Password: Error response from daemon: Get https://harbor.rainpole.com/v1/users/: \ x509: certificate signed by unknown authority (possibly because of \ "crypto/rsa: verification error" while trying to verify candidate \ authority certificate "Pivotal") cormac@pks-cli:~$
To resolve this first issue, I had to log into the Harbor UI as the Admin user. From, there I navigated to Administration > Configuration > System Settings, and then I clicked on the Download link associated with the Registry Root Cert, as shown below.
On my Ubuntu VM, the certificate needed to be placed in a particular directory /etc/docker/certs.d/harbor.rainpole.com, where harbor.rainpole.com is obviously the name of my registry that I am trying to login to. With the cert in place, I can now login to my registry, as shown below.
cormac@pks-cli:/etc/docker/certs.d/harbor.rainpole.com$ uname -a Linux pks-cli.rainpole.com4.13.0-46-generic #51-Ubuntu SMP Tue Jun 12 \ 12:36:29 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux cormac@pks-cli:/etc/docker/certs.d/harbor.rainpole.com$ ls ca.crt cormac@pks-cli:~$ sudo docker login -u admin harbor.rainpole.com Password: Login Succeeded
Cool. At this point, I thought I had solved the certificate issue. I was able to login to Harbor, tag images and push/pull to/from the registry. My next step was to deploy a couchbase app on my Kubernetes cluster, the image of which I had pushed to my registry. However, I got the following issue during the application creation:
root@pks-cli:~/cns-demo# kubectl get pods NAME READY STATUS RESTARTS AGE couchbase-0 0/1 ErrImagePull 0 12s root@pks-cli:~/cns-demo# kubectl describe pods Name: couchbase-0 Namespace: default Priority: 0 ... Events: Type Reason Age From \ Message ---- ------ ---- ---- \ ------- Normal Scheduled 3s default-scheduler \ Successfully assigned default/couchbase-0 to 2e2478da-5a3f-4941-90b1-9410f2cebab2 Normal SuccessfulAttachVolume 2s attachdetach-controller \ AttachVolume.Attach succeeded for volume "pvc-b5eb9ff9-2f2b-11e9-805e-00505682e96b" Normal Pulling <invalid> kubelet, 2e2478da-5a3f-4941-90b1-9410f2cebab2 \ pulling image"harbor.rainpole.com/library/saturnism/couchbase:k8s-petset" Warning Failed <invalid> kubelet, 2e2478da-5a3f-4941-90b1-9410f2cebab2 \ Failed to pull image "harbor.rainpole.com/library/saturnism/couchbase:k8s-petset": rpc error:\ code = Unknown desc = Error response from daemon: Get https://harbor.rainpole.com/v2/: x509:\ certificate signed by unknown authority Warning Failed <invalid> kubelet, 2e2478da-5a3f-4941-90b1-9410f2cebab2 \ Error: ErrImagePull Normal BackOff <invalid> kubelet, 2e2478da-5a3f-4941-90b1-9410f2cebab2 \ Back-off pulling image "harbor.rainpole.com/library/saturnism/couchbase:k8s-petset" Warning Failed <invalid> kubelet, 2e2478da-5a3f-4941-90b1-9410f2cebab2 \ Error: ImagePullBackOff root@pks-cli:~/cns-demo#
After some investigation, I found that I missed a step of integrating Harbor with PKS. In a nutshell, I should have copied the contents of my Harbor Registry CA certificate (same certificate I downloaded to my VM) and add it to the BOSH’s list of Trusted Certificates under Security in the BOSH tile in Pivotal Ops Manager. Once I had added it and applied the changes, I was successfully able to deploy my application.
root@pks-cli:~/cns-demo# kubectl get pods NAME READY STATUS RESTARTS AGE couchbase-0 1/1 Running 0 50s root@pks-cli:~/cns-demo# kubectl describe pods Name: couchbase-0 Namespace: default Priority: ... Events: Type Reason Age From \ Message ---- ------ ---- ---- \ ------- Warning FailedScheduling 30s (x6 over 37s) default-scheduler \ pod has unbound immediate PersistentVolumeClaims (repeated 3 times) Normal Scheduled 30s default-scheduler \ Successfully assigned default/couchbase-0 to e47914d4-efa3-4087-87f1-f7feb665b324 Normal SuccessfulAttachVolume 28s attachdetach-controller \ AttachVolume.Attach succeeded for volume "pvc-8f84d30d-2f8b-11e9-a131-005056821e38" Normal Pulling 20s kubelet, e47914d4-efa3-4087-87f1-f7feb665b324 \ pulling image "harbor.rainpole.com/library/saturnism/couchbase:k8s-petset" Normal Pulled 7s kubelet, e47914d4-efa3-4087-87f1-f7feb665b324 \ Successfully pulled image "harbor.rainpole.com/library/saturnism/couchbase:k8s-petset" Normal Created 7s kubelet, e47914d4-efa3-4087-87f1-f7feb665b324 \ Created container Normal Started 7s kubelet, e47914d4-efa3-4087-87f1-f7feb665b324 \ Started container root@pks-cli:~/cns-demo#
