I decided to dedicate a post to taking care of the Enterprise PKS prerequisites when deploying on VMware Cloud Foundation, namely the creation of the various certificates needed for trusted communication between the Enterprise PKS components (Operations Manager, BOSH, PKS and Harbor) and the rest of the environment. Unfortunately, the official VCF 3.9 documentation is a little light on the subject, simply stating that you should ‘Generate CA-Signed Certificates for Operations Manager, BOSH Director, Enterprise PKS control plane, and Harbor Registry‘. Therefore I decided that since it took me a bit of time to get these certificates setup for PKS on VCF, I would show you how to put these in place before going ahead with the full Enterprise PKS deployment.
1. What certificates do you need to deploy Enterprise PKS?
In order to successfully deploy Enterprise PKS on a workload domain in VMware Cloud Foundation (VCF), you need the following certificates:
- A Certificate of Authority (CA) Root Certificate PEM
- An NSX-T Super User Principal Identity Certificate
- An NSX-T Super User Principal Identity Key
- A certificate (base64 encoded) for Operations Manager
- A private key for Operations Manager
- A certificate (base64 encoded) for the PKS Appliance
- A private key for the PKS Appliance
- A certificate (base64 encoded) for Harbor
- A private key for Harbor
Let’s deal with the easy one first, the NSX-T Super User.
2. NSX-T Super User Principal Identity Certificate and Key
Enterprise PKS uses a super user to create, modify, and delete objects in NSX-T. The official VCF 3.9 documentation provides a bash shell script which can be run to generate the key for you. The only prerequisite is that openssl is installed and in your $PATH.
3. Root CA, Certificates and Private Keys for PKS Components
OK – so what is the best way to generate these CA-signed certificates? Well in Getting started with VCF Part 3, we already went through the process of setting up a Windows CA (see step 7 of that post). We are going to use the same Windows CA to create certificates for the Enterprise PKS components. Unfortunately, the certificate generator (generate_certificate.sh) that we used on the SDDC Manager to create vRA certificates does not appear to have a specific option for PKS, as the PKS deployment seems to be looking for individual certificates and keys. I’m not sure why that is, or even if it could be used for PKS. This is something I am going to look into a bit more internally. The good news is that our VVD (VMware Validated Design) team has already put a CertGenVVD tool together to create individual certificates for us automatically.
3.1 Required Tools and Software
You will need access to your Windows CA server, and install the following components:
- CertGenVVD version 5.0.006 downloaded from VMware Knowledgebase article 70678. There are some earlier versions (KB 2146215) but this latest version is the version that worked for me.
- OpenSSL for Windows – I went with the self-install executable, specifically the Win32 OpenSSL v1.1.1d Light EXE. You will also need to add this to the PATH environment variable of the server.
Visual C++ Redistributable Packages for Visual Studio 2013 (required by OpenSSL)
With these components available, you can now create your certificates. Note that while it should be possible to run this on a remote host that is part of your domain, firewalls and other nuances could cause issues. This is why I am running this tool directly on my CA server.
Next, open a PowerShell window (Run as Administrator), and run Set-ExecutionPolicy Unrestricted as per the instructions in the KB article. Now run the tool by simply typing in the name of the script, in this case CertGenVVD-5.0.006.ps1.
On initial launch, you can validate that you can successfully communicate with the CA Server, and everything is working as expected. Simply select v from the list of menu options. It will check that openssl, is available and in the path, as well as ensuring the CA is functioning. Here is my validate output.
This output looks good. You should resolve any issues that may be reported in the validation output.
3.3 Create the Root CA and PKS Components Keys and Certs
Now you need to create a .csv file with the names of your Enterprise PKS components, namely Operations Manager, BOSH, the PKS appliance and Harbor. Here is an example of my .csv with the 4 entries. It is pretty straight-forward, with a description at the beginning, then the name and the the domain, followed by the folder name where the certificates will be stored.
Name,DNS1,DNS2,DNS3,DNS4,DNS5,DNS6,DNS7,DNS8,DNS9,DNS10,DNS11,DOMAIN,FileName Ops Manager,w01-opsmgr-01,,,,,,,,,,,rainpole.com,w01-opsmgr-01 Bosh,w01-bosh-01,,,,,,,,,,,rainpole.com,w01-bosh-01 PKS,w01-pks-01,,,,,,,,,,,rainpole.coml,w01-pks-01 Harbor,w01-harbor-01,,,,,,,,,,,rainpole.com,w01-harbor-01 ,,,,,,,,,,,,, ,,,,,,,,,,,,,
From the CertGenVVD Menu, select option 1 to generate certificate files using an online Microsoft Root CA. When prompted, provide the path to your .csv file with the Enterprise PKS entries above, then provide an Organizational Value, Organization Unit value, Location, State and Country – the usual information required by certificates. The key size can be left at the default 2048. Next, you will be prompted for a keystore password for the P12/PEM certs (this program creates a whole bunch of different certificate formats, as we will see shortly). Once you’ve done that, the certificates start getting created in their own folders, as shown here:
Continuing with the certificate creation (sorry – could not get it all on one screenshot):
Excellent. Now if you look in the directory where you ran the tool, you should see two new folders, one called ConfigFiles, and the other called SignedByMSCACerts. The ConfigFiles simply has some text files created from the .csv for each of the Enterprise PKS components, whilst SignedByMSCACerts has your actual certs.
The RootCA folder has both a Root64.cer Security Certificate file and a DER file. We require a RootCA in a PEM format (.pem) for deployment in a Workload Domain on VCF.
Each of the PKS components have a number of certificate formats, as well as private keys. We will need the Base 64 encoded/Certificate only encodes certificate with a .crt extension for deployment of Enterprise PKS in a Workload Domain on VCF.
3.3.1 Some nuances for VCF – Creating a Root CA PEM
Although all the certificates are present and correct, they are not yet in the format required when deploying Enterprise PKS on VCF. As mentioned, we need a .pem format Root CA. This is easily resolved using openssl:
C:\SignedByMSCACerts\RootCA> openssl x509 -in Root64.cer -out Root64.pem -outform PEM C:\SignedByMSCACerts\RootCA> dir 02/04/2020 11:36 AM <DIR> . 02/04/2020 11:36 AM <DIR> .. 02/04/2020 10:34 AM 873 Root64-der.der 02/04/2020 10:34 AM 1,258 Root64.cer 02/04/2020 11:36 AM 1,258 Root64.pem 3 File(s) 3,389 bytes 2 Dir(s) 91,290,419,200 bytes free C:\SignedByMSCACerts\RootCA>
3.3.2 Some nuances for VCF – Creating .crt for PKS components
The deployment of Enterprise PKS on a workload domain in VCF requires a .crt for each of the Enterprise PKS certificates. A .cer already exists, which is a base 64 encoded/certificate only format. Simply make a copy of this in each folder (Operations Manager, BOSH, PKS and Harbor) and rename the extensions from .cer to .crt.
3.3.3 Some nuances for VCF – Choosing the correct Private Key for PKS components
The final item to highlight is that there are two private keys in each Enterprise PKS component’s folder, one with a .key extension and the other with a -orig.key. My understanding is that the .key has been modified for use with vSphere, and is not an actual private key. Therefore, when it comes to Enterprise PKS deployments, the -orig.key format should be used for the private key. You are now ready to deploy Enterprise PKS.
4. Choosing the correct certificates and keys at deployment
Let’s finish this post with a look at what the selection will be when we get to that point in the PKS deployment in a workload domain on VMware Cloud Foundation. I’ll do a full Enterprise PKS deployment post shortly. Here is that part of the deployment wizard, fully populated. Note the .pem extension on the Root CA Certificate, the .crt extensions for the PKS components and the use of the -orig.key for the Private Key.
Finally, if all is correct, the validation in step 10 should pass without any warnings or errors, and should just show INFO.
We’re now ready to deploy Enterprise PKS on a workload domain in VMware Cloud Foundation. If you want to see some of the previous VCF setup and deployment steps, they are all available here. Kudos again to my colleague Brian O’Connell for help and guidance with the CertGenVVD tool. My understanding is that a future version of the tool will create certificates that can be consumed directly by Enterprise PKS deployment, without the need to make the modifications seen in steps 3.3.x.