Replacing Data Services Manager Provider Appliance Certificate

One of the key goals in Data Services Manager (DSM) 2.1 is to enhance security. To that end, we have made a number of improvements around certificate management. One improvement is to allow customers to replace the default certificate in the DSM Provider Appliance with their own custom certificate. There are numerous ways to create your own custom certificate. You could choose the very manual process of using the openssl command, or if you have access to a Kubernetes cluster, you could use a ClusterIssuer Certificate Management Service (cert-manager). If you have the vSphere IaaS Control Plane (formerly known as vSphere with Tanzu), there is a Certificate Manager Supervisor Service which does exactly this. Details on how to install and configure this Supervisor Service to generate a certificate are available here. If you do not have a vSphere IaaS Control Plane, but do have access to another Kubernetes distribution, more details about how to use the Certificate Manager in other K8s distros can be found at https://cert-manager.io/.

However, the process of generating the certificate is outside the scope of this blog post. Instead, I wanted to go deeper into the steps to use your own custom cert as outlined in the official DSM 2.1 documentation. This should be useful to those of you who are still ramping up on Kubernetes.

Step 1: Download the DSM API Kubeconfig

The first step is to download the kubeconfig so that you can access the DSM API server. This is available by logging into the DSM UI, and in the top right hand corner, clicking on the user icon. This provides a drop-down menu where one of the options is to download the DSM API kubeconfig. Once the kubeconfig is downloaded, we can use it to query the current set of secrets, as well as create a new secret which contains our certificate information. Note that this is the kubeconfig downloaded via the DSM UI, and not the kubeconfig downloaded from the vSphere Client. Both configs have different privileges associated with them, and for this step, we definitely need the privileges associated with the DSM Admin.

Step 2: Create the Certificate “secret” YAML manifest

As per the official documentation, we need to create a secret that contains three pieces of information.

  • tls.crt: PEM signed certificate chain
  • tls.key: PEM private key
  • ca.crt: PEM CA certificate

These should then be added to a new Kubernetes “secret” object by adding them to a YAML manifest. This would look similar to the YAML manifest shown below, with the ca.crt, tls.key and tls.crt populated with your own leaf certificate, private key and certificate authority (CA) respectively. Full details on how to create a TLS secret can be found here. Note that the namespace in the manifest is set to dsm-system. You can choose a name of your choice.

apiVersion: v1
kind: Secret
metadata:
  name: provider-cert-secret
  namespace: dsm-system
type: kubernetes.io/tls
data:
  ca.crt: |
    <PEM CA certificate>
  tls.key: |
    <PEM private key>
  tls.crt: |
    <PEM signed certificate chain>

Note that if, rather than manually creating a certificate,  you used a Kubernetes Certificate Manager to create a Certificate Signing Request, the resulting certificate secret may need some ‘cleaning up’ before it can used for DSM. Typically, the resulting certificate/secret manifest will need the following entries removed:

  • creationTimesteamp
  • uid
  • resourceVersion
  • namespace (provider certificate secrets go into dsm-system namespace)

Now, we can create the new secret using the following kubectl commands. First, we look at the existing secrets. Then we create the new secret, and finally verify that it is present.

% kubectl get secrets -n dsm-system         
NAME                                        CREATED AT
db-backups-secret-tw0nj                     2024-07-23T14:05:09Z
db-backups-secret-twm6c                     2024-07-22T08:46:34Z
default-provider-backup-repo-secret-2bb56   2024-07-22T08:45:13Z
default-provider-backup-repo-secret-cvlhp   2024-07-23T14:30:18Z
default-provider-log-repo-secret-0tk7i      2024-07-22T08:46:08Z
ldap-default-secret-qq5k7                   2024-07-22T12:10:24Z
            
% kubectl apply -f my-cert-tls-secret.yaml -n dsm-system
secret/provider-cert-secret created

% kubectl get secrets -n dsm-system                     
NAME                                        CREATED AT
db-backups-secret-tw0nj                     2024-07-23T14:05:09Z
db-backups-secret-twm6c                     2024-07-22T08:46:34Z
default-provider-backup-repo-secret-2bb56   2024-07-22T08:45:13Z
default-provider-backup-repo-secret-cvlhp   2024-07-23T14:30:18Z
default-provider-log-repo-secret-0tk7i      2024-07-22T08:46:08Z
ldap-default-secret-qq5k7                   2024-07-22T12:10:24Z
provider-cert-secret                        2024-08-12T09:04:32Z

 Step 3: Updated the dsm-system-config with new cert secret

This is the final step. The certificate has been successfully created and wrapped in a Kubernetes secret object. We must now update the DSM configuration to use the new certificate by telling it about the secret. The DSM configuration is held in a custom resource called dsm-system-config. You can examine the current contents using the following commands (I’ve truncated the output below for brevity).

% kubectl get dsmcfg 
NAME                ALERTLEVEL   ROOTUSEREXPIRYDATE
dsm-system-config   ok           <invalid>

% kubectl describe dsmcfg dsm-system-config
Name:         dsm-system-config
Namespace:    
Labels:       dsm.vmware.com/trust-config=trusted-root-ca
Annotations:  dsm.vmware.com/dsm-dns-server-config: 2997244728867137537
              dsm.vmware.com/dsm-log-forwarding-config: 3650671831360427651
              dsm.vmware.com/observed-annotations:
                {"dsm.vmware.com/dsm-dns-server-config":"2997244728867137537","dsm.vmware.com/dsm-log-forwarding-config":"3650671831360427651","dsm.vmware...
              dsm.vmware.com/trust-config-version: dsm-system/trusted-root-ca/853
API Version:  system.dataservices.vmware.com/v1alpha1
Kind:         DsmSystemConfig
Metadata:
  Creation Timestamp:  2024-07-22T08:11:52Z
  Finalizers:
    dsmsystemsettings.system.dataservices.vmware.com/finalizer
  Generation:        3
  Resource Version:  20023
  UID:               2f38a715-8312-4c01-9139-0b4b90f07b24
Spec:
  Ceip Consent:  true
  Dns Names:
    dsm.rainpole.com
  Dns Servers:
<--- snip! --->

Now we need to update the Spec section, and add a new entry called spec.tls.secretName. To do this, we can use the command kubectl edit dsmcfg dsm-system-config and modify the spec section to add the new field which points to our certificate secret as follows:

  generation: 3
  labels:
    dsm.vmware.com/trust-config: trusted-root-ca
  name: dsm-system-config
  resourceVersion: "20023"
  uid: 2f38a715-8312-4c01-9139-0b4b90f07b24
spec:
  ceipConsent: true
  tls:
    secretName: provider-cert-secret
  dnsNames:
  - dsm.rainpole.com
  dnsServers:
<--- snip! --->

Now exit the editor, saving the changes you made to the dsm-system-config. If there are no reported errors after exiting, the new custom certificate is successfully in place. If there are errors reported, e.g., something similar to what is shown below, then it is possible that the certificate has not been created successfully, so you will have to repeat the steps outlined in this post and try again. It may even be that the leaf certificate, CA or private key are not created correctly in the first place, so revisit those steps as well.

% kubectl edit dsmcfg dsm-system-config
error: dsmsystemconfigs.system.dataservices.vmware.com "dsm-system-config" could not be patched: \
admission webhook "dsmsystemconfigs.system.dataservices.vmware.com" denied the request: \
error while validating certificates in secret dsm-system/provider-cert-secret: tls: \
failed to find any PEM data in key input

If the edit command returns without error, you have now successfully completed the updating of the default certificate in the DSM Provider Appliance with your own custom certificate.

Note: If you are using the latest DSM Custom Resources for Aria Automation, you will have to include the Certificate Authority (CA) used to create the DSM certificate and include it in the config.json if you wish to have secure communication between DSM and Aria Automation. Once added to the config.json, rerun the Python script to upload the CA. The CA can be found in the secret you built to request the certificate.

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.