DSM + MinIO: Certificate chain obtained from endpoint is incomplete or empty

I recently worked with one of our customers to configure MinIO object storage on-premises.  The plan was to create a number of S3 compatible buckets to provide image, backup and log repositories for VMware Data Services Manager v2.0.x. For security reasons, the customer wanted to create an intermediate certificate of authority (CA), and use that to sign the leaf certificates rather than sign them with a root CA. This is common practice. However, the customer hit the above issue when trying to use the MinIO object store buckets with the leaf certs created from the intermediate cert. I decided to try to reproduce the issue in-house and attempt to resolve it. To achieve a similar setup, I followed this article on how to create a self-signed root certificate and an intermediate certificate. The intermediate certificate was then used to sign the leaf certificate for my MinIO server. I then proceeded to follow the MinIO documentation and placed the root CA and intermediate CA (concatenated into a single file) in the following directory:

${HOME}/.minio/certs/CAs

The leaf (public) certificate was placed in the following directory:

${HOME}/.minio./certs/public.crt

I then ran the following command to verify that the certificate chain was indeed working as expected (the root CA and the intermediate CA were both concatenated into the intermediateCA.crt file).

$ openssl verify -CAfile ${HOME}/.minio/certs/CAs/intermediateCA.crt ${HOME}/.minio/certs/public.crt
public.crt: OK

However, when I tried to run a test from DSM to the MinIO server, some problems were seen. First, when trying to add the bucket to the DSM configuration, I saw the following error:

Next, when I tried to do a test using openssl from the DSM appliance to the MinIO server, I saw this:

# openssl s_client -connect xx.xx.xx.xx:9000
CONNECTED(00000003)
Can't use SSL_get_servername
depth=0 C = IE, ST = Cork, O = VCF, CN = xx.xx.xx.xx
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 C = IE, ST = Cork, O = VCF, CN = xx.xx.xx.xx
verify error:num=21:unable to verify the first certificate
verify return:1
depth=0 C = IE, ST = Cork, O = VCF, CN = xx.xx.xx.xx
verify return:1
---
Certificate chain
 0 s:C = IE, ST = Cork, O = VCF, CN = xx.xx.xx.xx
   i:C = IE, ST = Cork, O = VCF, CN = Minio Intermediate CA
   a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
   v:NotBefore: May 17 10:27:43 2024 GMT; NotAfter: May 27 10:27:43 2025 GMT
---

At that point, I received some advice from our engineering team to concatenate all of the certificates (root, intermediate and leaf) into a single .minio/certs/public.crt file. This is what I did, and this action allowed me to successfully use the MinIO server as an object store for the S3 compatible buckets required by DSM. I also have a much better test result between DSM and the MinIO server.

# openssl s_client -connect xx.xx.xx.xx:9000 
CONNECTED(00000003)
Can't use SSL_get_servername
depth=2 CN = Minio Root CA, C = IE, ST = Cork, O = VCF
verify error:num=19:self-signed certificate in certificate chain
verify return:1
depth=2 CN = Minio Root CA, C = IE, ST = Cork, O = VCF
verify return:1
depth=1 C = IE, ST = Cork, O = VCF, CN = Minio Intermediate CA
verify return:1
depth=0 C = IE, ST = Cork, O = VCF, CN = xx.xx.xx.xx
verify return:1
---
Certificate chain
 0 s:C = IE, ST = Cork, O = VCF, CN = xx.xx.xx.xx
   i:C = IE, ST = Cork, O = VCF, CN = Minio Intermediate CA
   a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
   v:NotBefore: May 17 10:27:43 2024 GMT; NotAfter: May 27 10:27:43 2025 GMT
 1 s:C = IE, ST = Cork, O = VCF, CN = Minio Intermediate CA
   i:CN = Minio Root CA, C = IE, ST = Cork, O = VCF
   a:PKEY: rsaEncryption, 4096 (bit); sigalg: RSA-SHA256
   v:NotBefore: May 17 10:21:44 2024 GMT; NotAfter: May 15 10:21:44 2034 GMT
 2 s:CN = Minio Root CA, C = IE, ST = Cork, O = VCF
   i:CN = Minio Root CA, C = IE, ST = Cork, O = VCF
   a:PKEY: rsaEncryption, 4096 (bit); sigalg: RSA-SHA256
   v:NotBefore: May 17 10:18:22 2024 GMT; NotAfter: May 12 10:18:22 2044 GMT
---

 In conclusion, I am not completely sure why I needed to concat all of the certificates into a single public.crt to make this work. It could be because this was a self-signed root CA, and not a well-known root CA. Usually with well known CAs, we only need to put the leaf certificate into the public.crt. The chain gets validated and everything (browser, API client) works. Anyway, whatever the case, if you do come across this “Certificate chain obtained from endpoint is incomplete or empty” using an intermediate certificate with Data Services Manager and MinIO, placing all the certificates (root, intermediate and leaf) into the public.crt file should help.