Reviewing PKS logs and status

After a bit of a sabbatical, I am back to looking PKS (Pivotal Container Service) again. I wanted to look at the new version 1.3, but I had to do a bit of work on my environment to allow me to do this. Primarily, I needed to upgrade my NSX-T environment from version 2.1 to 2.3. I followed this blog post from vmtechie which provides a useful step-by-step guide. Kudos to our VMware NSX-T team as the upgrade worked without a hitch. My next step was to start work on the PKS deployment. I just did a brand new deployment of PKS 1.3. I used the following set of blogs post from Keith Lee which provided invaluable help. Now, because I have a weird lab network setup, I have my PKS components on one VLAN and my vSphere management components on another VLAN. This means I have had to multi-homed a number of pieces of infra such as my vCenter appliance, my NSX Manager and my PCF OPs Manager. This caused me a bit a grief when it came to configuring the certificates (to establish trust between BOSH and PKS and the NSX Manager). Fortunately, like all these things, I learnt a little bit more about PKS troubleshooting during this exercise and I’ve decided to share some of these tidbits with you in this post.

On a few occasions, my PKS deployment got as far as this point and then keeled over (This is the verbose log output from the PKS (Pivotal Container Service) Tile by the way):

Now, to actually do further troubleshooting, I can use the following command to open an SSH session to the VM (-e specifies the environment and -d the deployment; the last argument is the instance):

Now that we are in the PKS VM shell, we can become superuser, and query the state of the processes/jobs using the “monit summary” command. As you can see below, there is an issue with pks-nsx-t-osb-proxy which was also reported in output above. I read a few community posts that already stated that this job has an issue if certificates haven’t been setup correctly. Let’s see if we can figure out what the problem is.