Replacing Data Services Manager Provider Appliance Certificate
One of the key goals in Data Services Manager (DSM) 2.1 is to enhance security. To that end, we have made a number of improvements around certificate management. One improvement is to allow customers to replace the default certificate in the DSM Provider Appliance with their own custom certificate. There are numerous ways to create your own custom certificate. You could choose the very manual process of using the openssl command, or if you have access to a Kubernetes cluster, you could use a ClusterIssuer Certificate Management Service (cert-manager). If you have the vSphere IaaS Control Plane (formerly known as vSphere with Tanzu), there is a Certificate Manager Supervisor Service which does exactly this. Details on how to install and configure this Supervisor Service to generate a certificate are available here. If you do not have a vSphere IaaS Control Plane, but do have access to another Kubernetes distribution, more details about how to use the Certificate Manager in other K8s distros can be found at https://cert-manager.io/.
However, the process of generating the certificate is outside the scope of this blog post. Instead, I wanted to go deeper into the steps to use your own custom cert as outlined in the official DSM 2.1 documentation. This should be useful to those of you who are still ramping up on Kubernetes.
Step 1: Download the DSM API Kubeconfig
The first step is to download the kubeconfig so that you can access the DSM API server. This is available by logging into the DSM UI, and in the top right hand corner, clicking on the user icon. This provides a drop-down menu where one of the options is to download the DSM API kubeconfig. Once the kubeconfig is downloaded, we can use it to query the current set of secrets, as well as create a new secret which contains our certificate information. Note that this is the kubeconfig downloaded via the DSM UI, and not the kubeconfig downloaded from the vSphere Client. Both configs have different privileges associated with them, and for this step, we definitely need the privileges associated with the DSM Admin.
Step 2: Create the Certificate “secret” YAML manifest
As per the official documentation, we need to create a secret that contains three pieces of information.
- tls.crt: PEM signed certificate chain
- tls.key: PEM private key
- ca.crt: PEM CA certificate
These should then be added to a new Kubernetes “secret” object by adding them to a YAML manifest. This would look similar to the YAML manifest shown below, with the ca.crt, tls.key and tls.crt populated with your own leaf certificate, private key and certificate authority (CA) respectively. Full details on how to create a TLS secret can be found here. Note that the namespace in the manifest is set to dsm-system. You can choose a name of your choice.
apiVersion: v1 kind: Secret metadata: name: provider-cert-secret namespace: dsm-system type: kubernetes.io/tls data: ca.crt: | <PEM CA certificate> tls.key: | <PEM private key> tls.crt: | <PEM signed certificate chain>
Note that if, rather than manually creating a certificate, you used a Kubernetes Certificate Manager to create a Certificate Signing Request, the resulting certificate secret may need some ‘cleaning up’ before it can used for DSM. Typically, the resulting certificate/secret manifest will need the following entries removed:
- creationTimesteamp
- uid
- resourceVersion
- namespace (provider certificate secrets go into dsm-system namespace)
Now, we can create the new secret using the following kubectl commands. First, we look at the existing secrets. Then we create the new secret, and finally verify that it is present.
% kubectl get secrets -n dsm-system NAME CREATED AT db-backups-secret-tw0nj 2024-07-23T14:05:09Z db-backups-secret-twm6c 2024-07-22T08:46:34Z default-provider-backup-repo-secret-2bb56 2024-07-22T08:45:13Z default-provider-backup-repo-secret-cvlhp 2024-07-23T14:30:18Z default-provider-log-repo-secret-0tk7i 2024-07-22T08:46:08Z ldap-default-secret-qq5k7 2024-07-22T12:10:24Z % kubectl apply -f my-cert-tls-secret.yaml -n dsm-system secret/provider-cert-secret created % kubectl get secrets -n dsm-system NAME CREATED AT db-backups-secret-tw0nj 2024-07-23T14:05:09Z db-backups-secret-twm6c 2024-07-22T08:46:34Z default-provider-backup-repo-secret-2bb56 2024-07-22T08:45:13Z default-provider-backup-repo-secret-cvlhp 2024-07-23T14:30:18Z default-provider-log-repo-secret-0tk7i 2024-07-22T08:46:08Z ldap-default-secret-qq5k7 2024-07-22T12:10:24Z provider-cert-secret 2024-08-12T09:04:32Z
Step 3: Updated the dsm-system-config with new cert secret
This is the final step. The certificate has been successfully created and wrapped in a Kubernetes secret object. We must now update the DSM configuration to use the new certificate by telling it about the secret. The DSM configuration is held in a custom resource called dsm-system-config. You can examine the current contents using the following commands (I’ve truncated the output below for brevity).
% kubectl get dsmcfg NAME ALERTLEVEL ROOTUSEREXPIRYDATE dsm-system-config ok <invalid> % kubectl describe dsmcfg dsm-system-config Name: dsm-system-config Namespace: Labels: dsm.vmware.com/trust-config=trusted-root-ca Annotations: dsm.vmware.com/dsm-dns-server-config: 2997244728867137537 dsm.vmware.com/dsm-log-forwarding-config: 3650671831360427651 dsm.vmware.com/observed-annotations: {"dsm.vmware.com/dsm-dns-server-config":"2997244728867137537","dsm.vmware.com/dsm-log-forwarding-config":"3650671831360427651","dsm.vmware... dsm.vmware.com/trust-config-version: dsm-system/trusted-root-ca/853 API Version: system.dataservices.vmware.com/v1alpha1 Kind: DsmSystemConfig Metadata: Creation Timestamp: 2024-07-22T08:11:52Z Finalizers: dsmsystemsettings.system.dataservices.vmware.com/finalizer Generation: 3 Resource Version: 20023 UID: 2f38a715-8312-4c01-9139-0b4b90f07b24 Spec: Ceip Consent: true Dns Names: dsm.rainpole.com Dns Servers: <--- snip! --->
Now we need to update the Spec section, and add a new entry called spec.tls.secretName. To do this, we can use the command kubectl edit dsmcfg dsm-system-config and modify the spec section to add the new field which points to our certificate secret as follows:
generation: 3 labels: dsm.vmware.com/trust-config: trusted-root-ca name: dsm-system-config resourceVersion: "20023" uid: 2f38a715-8312-4c01-9139-0b4b90f07b24 spec: ceipConsent: true tls: secretName: provider-cert-secret dnsNames: - dsm.rainpole.com dnsServers: <--- snip! --->
Now exit the editor, saving the changes you made to the dsm-system-config. If there are no reported errors after exiting, the new custom certificate is successfully in place. If there are errors reported, e.g., something similar to what is shown below, then it is possible that the certificate has not been created successfully, so you will have to repeat the steps outlined in this post and try again. It may even be that the leaf certificate, CA or private key are not created correctly in the first place, so revisit those steps as well.
% kubectl edit dsmcfg dsm-system-config error: dsmsystemconfigs.system.dataservices.vmware.com "dsm-system-config" could not be patched: \ admission webhook "dsmsystemconfigs.system.dataservices.vmware.com" denied the request: \ error while validating certificates in secret dsm-system/provider-cert-secret: tls: \ failed to find any PEM data in key input
If the edit command returns without error, you have now successfully completed the updating of the default certificate in the DSM Provider Appliance with your own custom certificate.
Note: If you are using the latest DSM Custom Resources for Aria Automation, you will have to include the Certificate Authority (CA) used to create the DSM certificate and include it in the config.json if you wish to have secure communication between DSM and Aria Automation. Once added to the config.json, rerun the Python script to upload the CA. The CA can be found in the secret you built to request the certificate.