Setting up Nginx reverse-proxy for distributed Minio S3 deployment

I wanted to follow-up on my recent Minio S3 post with steps on how to implement a reverse-proxy using Nginx. The purpose of this is to allow an end-user to connect to a single Minio server, and have that connection be redirected in a round-robin fashion to all of my other 16 Minio servers in my Minio S3 deployment. This was surprisingly very straight-forward, and only required a handful of changes to my nginx.conf file. If you want to review the initial deployment steps, you can find these here in my original post. Let’s go through the steps to set up a reverse-proxy load-balancer for Minio S3 using Nginx next.

Step 1 – Deploy Nginx

I deployed Nginx on my first Minio server, minio1. This was also the server where I had installed my Minio client, mc. Before installing nginx, I first needed to deploy EPEL (Extra Packages for Enterprise Linux). EPEL is a repository of high-quality add-on packages that complement the Fedora-based Red Hat Enterprise Linux (RHEL) and its compatible spinoffs, such as CentOS .

  •  sudo yum install epel-release
  •  sudo yum install nginx

 

Step 2 – Enable & Verify Nginx

To enable and start nginx on Centos, there are only 2 commands needed.

  •  systemctl enable nginx
  •  systemctl start nginx

 

Now if you point a browser to the external IP address or FQDN of the VM, you should get a default nginx landing page.

Step 3 – Configure the /etc/nginx/nginx.conf file to redirect to Minio S3

Now, the next step is to have this landing page automatically redirect to the Minio S3 object store on port 9000. To do this, we must edit the /etc/nginx/nginx.conf and add some proxy entries into the location{} stanza.

First, make a copy of the nginx.conf before editing:

  • cd /etc/nginx
  • cp nginx.conf nginx.conf.orig

Next, make the following changes using the editor of your choice:

from:

location / {
}

to:

location / {
proxy_set_header Host $http_host;
proxy_pass https://localhost:9000;
}

Note that I have used https in proxy_pass above since I have enabled access via SSL to my Minio object store in my previous setup. If you have not done this, then you can simply use http. Now, save your changes and restart nginx:

  •  systemctl restart nginx

If you are having issues restarting nginx after making the edits, use nginx -t to check the syntax of the nginx.conf. It is extremely useful for checking typos, formatting, etc. Here are some examples:

# nginx -t
nginx: [emerg] unexpected “}” in /etc/nginx/nginx.conf:50
nginx: configuration file /etc/nginx/nginx.conf test failed

# nginx -t
nginx: [emerg] invalid URL prefix in /etc/nginx/nginx.conf:49
nginx: configuration file /etc/nginx/nginx.conf test failed

# nginx -t
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful

 

Now point your browser to the IP address or FQDN of this VM, and it should automatically redirect to Minio S3 browser. Success!

 

Step 4 – Configure the /etc/nginx/nginx.conf file to redirect to all servers in a round robin

OK, so now we have simple redirection running. But how do we get this nginx front-end to proxy requests to all of our Minio servers in a round-robin fashion? Nginx can do this for us as well. This time, we need to make a list of all our servers, and place them into the http{} stanza, just before the server{} stanza. The format for this upstream {} stanza is upstream <some_label> {}.

upstream minio_servers {
server minio1:9000;
server minio2:9000;
server minio3:9000;
server minio4:9000;
server minio5:9000;
server minio6:9000;
server minio7:9000;
server minio8:9000;
server minio9:9000;
server minio10:9000;
server minio11:9000;
server minio12:9000;
server minio13:9000;
server minio14:9000;
server minio15:9000;
server minio16:9000;
}

And now we also need to make an update to the location{} stanza. Whereas before the proxy_pass pointed to https://localhost:9000, we now make it point to the list of upstream minio servers above, as follows:

proxy_pass https://minio_servers;

Note the name used in proxy_pass must match the name of the upstream label. Save the nginx.conf file. Verify that it is syntactically correct with nginx -t, and then restart the nginx service.

# nginx -t
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful

# systemctl restart nginx

You should once again be able to access the Minio S3 browser by simply putting in the IP address or FQDN of the VM, but this time requests are redirected among all the servers in the Minio distributed deployment. Pretty cool.

 

Step 5: Use SSL/port 443 for access

I mentioned previously that I had set up SSL so that I could securely access the Minio S3 browser, rather than use http port 80. So how do I force that in the nginx.conf file? In the nginx.conf file, there is another server{} stanza that is commented out. Some sample configuration files refer to this as the “Settings for a TLS enabled server”. Others just comment it as “HTTPS server”. Either way, they are both the same. What we are going to do is create a second server {} stanza to handle requests on port 443/https.

These are the lines you need to uncomment and/or modify in this server{} stanza.

server {
    listen 443 ssl http2 default_server;
    listen [::]:443 ssl http2 default_server;
    server_name _;

       ssl_certificate “/root/.minio/certs/public.crt”;
       ssl_certificate_key “/root/.minio/certs/private.key”;

       location / {
          proxy_set_header Host $http_host;
          proxy_pass https://minio_servers;

    }
}

The ssl_certificate and the ssl_certificate_key, once un-commented, need to be updated with the path to the public certificate and private key. These are the cert and key that were created to enable encryption in my previous post. The location {} stanza entries for proxy_set_header and proxy_pass are the same as before, and will reference each of the minio servers in a round robin fashion. At this point, save the nginx.conf, check it with nginx -t and finally restart the nginx service. Now when we point our browser using https to the IP address or FQDN of the VM, we should get secure access.

You can also remove the first server {} stanza for port 80, so that all access is now via https.

Something to note. Since Centos 7 has SE Linux extensions, it won’t allow Nginx access to files under /root automatically. If SE Linux is not completely disabled, then you might see the following on trying to start nginx:

# systemctl restart nginx
Job for nginx.service failed because the control process exited with error code. See “systemctl status nginx.service” and “journalctl -xe” for details.

 

If you examine the log output using journalctl -xe, you will see something like the following reported.

Jul 03 11:16:20 minio3.rainpole.com setroubleshoot[15368]: Deleting alert 1a8d840d-c2be-4541-bc8b-756c0987509d, it is allowed in cur
Jul 03 11:16:20 minio3.rainpole.com setroubleshoot[15368]: SELinux is preventing /usr/sbin/nginx from open access on the file /root/
Jul 03 11:16:20 minio3.rainpole.com python[15368]: SELinux is preventing /usr/sbin/nginx from open access on the file /root/.minio/c

***** Plugin catchall (100. confidence) suggests **************************

If you believe that nginx should be allowed open access on the public.crt file by
Then you should report this as a bug.
You can generate a local policy module to allow this access.
Do
allow this access for now by executing:
# ausearch -c ‘nginx’ –raw | audit2allow -M my-nginx
# semodule -i my-nginx.pp

 

You can either move the files to somewhere that SELinux allows nginx to access, disable SELinux completely, or just run to the two commands above in the output above. You will have to repeat this twice, once for the public.crt, and again for the private.key. Once I ran these commands twice, I was successfully able to restart the nginx server.

So there you go – we can now use Nginx as a front-end proxy for load balancing S3 requests across all of our Minio servers deployed on-prem on top of vSAN. Hopefully this will help if you need to go through this process. Minio also provide a doc on additional parameters and settings by clicking here.

 

Step 6 – How can you tell if load balancing is working?

This was a follow-up question, and it was a good one. How could you actually tell if the requests were being successfully upstreamed across the different servers. To do this, I made a new log_format entry with some specific upstream entries in the http{} stanza in nginx.conf as follows:

log_format main_ext ‘$remote_addr – $remote_user [$time_local] “$request” ‘
‘$status $body_bytes_sent “$http_referer” ‘
‘”$http_user_agent” “$http_x_forwarded_for” ‘
‘”$host” sn=”$server_name” ‘ ‘rt=$request_time ‘
‘ua=”$upstream_addr” us=”$upstream_status” ‘
‘ut=”$upstream_response_time” ul=”$upstream_response_length” ‘
‘cs=$upstream_cache_status’;

 

Now I modified the access_log entry to use this new format.

access_log /var/log/nginx/access.log main_ext;

 

And now when I check the access log after launching the Minio S3 browser, I can see references to multiple upstream addresses. LGTM.

10.30.3.220 – – [04/Jul/2018:12:58:25 +0100] “GET /minio/loader.css HTTP/2.0” 200 504 “https://10.27.51.47/minio/” “Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:60.0) Gecko/20100101 Firefox/60.0” “-” “10.27.51.47” sn=”_” rt=0.007 ua=”10.27.51.47:9000″ us=”200″ ut=”0.007″ ul=”504″ cs=-
10.30.3.220 – – [04/Jul/2018:12:58:25 +0100] “GET /minio/logo.svg HTTP/2.0” 200 1034 “https://10.27.51.47/minio/” “Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:60.0) Gecko/20100101 Firefox/60.0” “-” “10.27.51.47” sn=”_” rt=0.011 ua=”10.27.51.49:9000″ us=”200″ ut=”0.011″ ul=”1034″ cs=-
10.30.3.220 – – [04/Jul/2018:12:58:25 +0100] “POST /minio/webrpc HTTP/2.0” 200 706 “https://10.27.51.47/minio/” “Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:60.0) Gecko/20100101 Firefox/60.0” “-” “10.27.51.47” sn=”_” rt=0.007 ua=”10.27.51.34:9000″ us=”200″ ut=”0.007″ ul=”706″ cs=-
10.30.3.220 – – [04/Jul/2018:12:58:25 +0100] “POST /minio/webrpc HTTP/2.0” 200 564 “https://10.27.51.47/minio/” “Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:60.0) Gecko/20100101 Firefox/60.0” “-” “10.27.51.47” sn=”_” rt=0.018 ua=”10.27.51.35:9000″ us=”200″ ut=”0.005″ ul=”564″ cs=-
10.30.3.220 – – [04/Jul/2018:12:58:25 +0100] “POST /minio/webrpc HTTP/2.0” 200 1799 “https://10.27.51.47/minio/” “Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:60.0) Gecko/20100101 Firefox/60.0” “-” “10.27.51.47” sn=”_” rt=0.123 ua=”10.27.51.56:9000″ us=”200″ ut=”0.123″ ul=”1799″ cs=-