Getting started with VCF Part 8 – PKS Certificates

I decided to dedicate a post to taking care of the Enterprise PKS prerequisites when deploying on VMware Cloud Foundation, namely the creation of the various certificates needed for trusted communication between the Enterprise PKS components (Operations Manager, BOSH, PKS and Harbor) and the rest of the environment. Unfortunately, the official VCF 3.9 documentation is a little light on the subject, simply stating that you should ‘Generate CA-Signed Certificates for Operations Manager, BOSH Director, Enterprise PKS control plane, and Harbor Registry‘. Therefore I decided that since it took me a bit of time to get these certificates setup for PKS on VCF, I would show you how to put these in place before going ahead with the full Enterprise PKS deployment.

1. What certificates do you need to deploy Enterprise PKS?

In order to successfully deploy Enterprise PKS on a workload domain in VMware Cloud Foundation (VCF), you need the following certificates:

  1. A Certificate of Authority (CA) Root Certificate PEM
  2. An NSX-T Super User Principal Identity Certificate
  3. An NSX-T Super User Principal Identity Key
  4. A certificate (base64 encoded) for Operations Manager
  5. A private key for Operations Manager
  6. A certificate (base64 encoded) for the PKS Appliance
  7. A private key for the PKS Appliance
  8. A certificate (base64 encoded) for Harbor
  9. A private key for Harbor

Let’s deal with the easy one first, the NSX-T Super User.

2. NSX-T Super User Principal Identity Certificate and Key

Enterprise PKS uses a super user to create, modify, and delete objects in NSX-T. The official VCF 3.9 documentation provides a bash shell script which can be run to generate the key for you. The only prerequisite is that openssl is installed and in your $PATH.

3. Root CA, Certificates and Private Keys for PKS Components

OK – so what is the best way to generate these CA-signed certificates? Well in Getting started with VCF Part 3, we already went through the process of setting up a Windows CA (see step 7 of that post). We are going to use the same Windows CA to create certificates for the Enterprise PKS components. Unfortunately, the certificate generator (generate_certificate.sh) that we used on the SDDC Manager to create vRA certificates does not appear to have a specific option for PKS, as the PKS deployment seems to be looking for individual certificates and keys. I’m not sure why that is, or even if it could be used for PKS. This is something I am going to look into a bit more internally. The good news is that our VVD (VMware Validated Design) team has already put a CertGenVVD tool together to create individual certificates for us automatically.

3.1 Required Tools and Software

You will need access to your Windows CA server, and install the following components:

With these components available, you can now create your certificates. Note that while it should be possible to run this on a remote host that is part of your domain, firewalls and other nuances could cause issues. This is why I am running this tool directly on my CA server.

Next, open a PowerShell window (Run as Administrator), and run Set-ExecutionPolicy Unrestricted as per the instructions in the KB article. Now run the tool by simply typing in the name of the script, in this case CertGenVVD-5.0.006.ps1.

3.2 Validation

On initial launch, you can validate that you can successfully communicate with the CA Server, and everything is working as expected. Simply select v from the list of menu options. It will check that openssl, is available and in the path, as well as ensuring the CA is functioning. Here is my validate output.

This output looks good. You should resolve any issues that may be reported in the validation output.

3.3 Create the Root CA and PKS Components Keys and Certs

Now you need to create a .csv file with the names of your Enterprise PKS components, namely Operations Manager, BOSH, the PKS appliance and Harbor. Here is an example of my .csv with the 4 entries. It is pretty straight-forward, with a description at the beginning, then the name and the the domain, followed by the folder name where the certificates will be stored.

Name,DNS1,DNS2,DNS3,DNS4,DNS5,DNS6,DNS7,DNS8,DNS9,DNS10,DNS11,DOMAIN,FileName
Ops Manager,w01-opsmgr-01,,,,,,,,,,,rainpole.com,w01-opsmgr-01
Bosh,w01-bosh-01,,,,,,,,,,,rainpole.com,w01-bosh-01
PKS,w01-pks-01,,,,,,,,,,,rainpole.coml,w01-pks-01
Harbor,w01-harbor-01,,,,,,,,,,,rainpole.com,w01-harbor-01
,,,,,,,,,,,,,
,,,,,,,,,,,,,

From the CertGenVVD Menu, select option 1 to generate certificate files using an online Microsoft Root CA. When prompted, provide the path to your .csv file with the Enterprise PKS entries above, then provide an Organizational Value, Organization Unit value, Location, State and Country – the usual information required by certificates. The key size can be left at the default 2048. Next, you will be prompted for a keystore password for the P12/PEM certs (this program creates a whole bunch of different certificate formats, as we will see shortly). Once you’ve done that, the certificates start getting created in their own folders, as shown here:

Continuing with the certificate creation (sorry – could not get it all on one screenshot):

Excellent. Now if you look in the directory where you ran the tool, you should see two new folders, one called ConfigFiles, and the other called SignedByMSCACerts. The ConfigFiles simply has some text files created from the .csv for each of the Enterprise PKS components, whilst SignedByMSCACerts has your actual certs.

The RootCA folder has both a Root64.cer Security Certificate file and a DER file. We require a RootCA in a PEM format (.pem) for deployment in a Workload Domain on VCF.

Each of the PKS components have a number of certificate formats, as well as private keys. We will need the Base 64 encoded/Certificate only encodes certificate with a .crt extension for deployment of Enterprise PKS in a Workload Domain on VCF.

3.3.1 Some nuances for VCF – Creating a Root CA PEM

Although all the certificates are present and correct, they are not yet in the format required when deploying Enterprise PKS on VCF. As mentioned, we need a .pem format Root CA. This is easily resolved using openssl:

C:\SignedByMSCACerts\RootCA> openssl x509 -in Root64.cer -out Root64.pem -outform PEM

C:\SignedByMSCACerts\RootCA> dir

02/04/2020  11:36 AM    <DIR>          .
02/04/2020  11:36 AM    <DIR>          ..
02/04/2020  10:34 AM               873 Root64-der.der
02/04/2020  10:34 AM             1,258 Root64.cer
02/04/2020  11:36 AM             1,258 Root64.pem
               3 File(s)          3,389 bytes
               2 Dir(s)  91,290,419,200 bytes free

C:\SignedByMSCACerts\RootCA>

3.3.2 Some nuances for VCF – Creating .crt for PKS components

The deployment of Enterprise PKS on a workload domain in VCF requires a .crt for each of the Enterprise PKS certificates. A .cer already exists, which is a base 64 encoded/certificate only format. Simply make a copy of this in each folder (Operations Manager, BOSH, PKS and Harbor) and rename the extensions from .cer to .crt.

3.3.3 Some nuances for VCF – Choosing the correct Private Key for PKS components

The final item to highlight is that there are two private keys in each Enterprise PKS component’s folder, one with a .key extension and the other with a -orig.key. My understanding is that the .key has been modified for use with vSphere, and is not an actual private key. Therefore, when it comes to Enterprise PKS deployments, the -orig.key format should be used for the private key. You are now ready to deploy Enterprise PKS.

4. Choosing the correct certificates and keys at deployment

Let’s finish this post with a look at what the selection will be when we get to that point in the PKS deployment in a workload domain on VMware Cloud Foundation. I’ll do a full Enterprise PKS deployment post shortly. Here is that part of the deployment wizard, fully populated. Note the .pem extension on the Root CA Certificate, the .crt extensions for the PKS components and the use of the -orig.key for the Private Key.

Finally, if all is correct, the validation in step 10 should pass without any warnings or errors, and should just show INFO.

We’re now ready to deploy Enterprise PKS on a workload domain in VMware Cloud Foundation. If you want to see some of the previous VCF setup and deployment steps, they are all available here. Kudos again to my colleague Brian O’Connell for help and guidance with the CertGenVVD tool. My understanding is that a future version of the tool will create certificates that can be consumed directly by Enterprise PKS deployment, without the need to make the modifications seen in steps 3.3.x.

7 Replies to “Getting started with VCF Part 8 – PKS Certificates”

  1. Hi Cormac, great blog!

    One thing we noticed deploying PKS for VCF on VxRail was that the bosh certificate was not required, this is managed internally by Ops Manager, you will notice that when you were going through the deployment that no request was made for the bosh certificate, thanks to my colleague Cliff Cahill for spotting that detail.

    One other important thing to note is that the In the deployment of PKS three VMs are deployed (By the Operations manager), and the first 3 IPs in the range after the exclusion list from the PKS management subnet are taken and assigned in this order.
    1. Bosh
    2. PKS
    3. Harbor – if selected

    User provides FQDN for harbor and PKS during deployment so the DNS entries need to be in the correct for the above entries, so based on the exclusion list provided in the mgmt network the IPs of the various components can be determined.

    1. Thanks Cathal – I actually hadn’t twigged that there was no request for the BOSH cert. Looks like I’ll have to make some edits to the post.

      I have the PKS deploy post coming out later today, which talks about the exclusion ranges and so on.

      1. Where do we need to paste the script? Is it in powershl or Linux machine?

        I am getting error when i tried to paste below script on Centos VM.

        [root@localhost ssl]# NSX_SUPERUSER_CERT_FILE=”pks-nsx-t-superuser.crt”
        [root@localhost ssl]# NSX_SUPERUSER_KEY_FILE=”pks-nsx-t-superuser.key”
        [root@localhost ssl]#
        [root@localhost ssl]# openssl req \
        > -newkey rsa:2048 \
        > -x509 \
        > -nodes \
        > -keyout “$NSX_SUPERUSER_KEY_FILE” \
        > -new \
        > -out “$NSX_SUPERUSER_CERT_FILE” \
        > -subj /CN=ocp-nsx-t-superuser \
        > -extensions client_server_ssl \
        > -config cat /etc/ssl/openssl.cnf \
        > ) \
        > -sha256 \
        > -days 730
        Generating a 2048 bit RSA private key
        …+++
        ……..+++
        writing new private key to ‘pks-nsx-t-superuser.key’
        —–
        unable to find ‘distinguished_name’ in config
        problems making Certificate Request
        140174808377232:error:0E06D06C:configuration file routines:NCONF_get_string:no value:conf_lib.c:324:group=req name=distinguished_name

        Am I missing anything like input parameters required for certs?

        1. The script you are using is not correct.

          Check the docs once more, but you need the following:

          cat /etc/ssl/openssl.cnf \
          <(printf '[client_server_ssl]\nextendedKeyUsage = clientAuth\n')

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.