Site icon CormacHogan.com

Validating overlay network when docker swarm running on Centos VMs on vSphere

Cormac

I got a chance to revisit my docker swarm deployment this week after a bit of a break. I was a little curious about my setup because when I spoke to some of our ‘Project Hatchway‘ engineers, I was told that I should be able to launch a single instance of Nginx in Docker Swarm (“docker service create –replicas 1 -p 8080:80 –name web nginx”) and I should be able to access the web service using the following command from any swarm node – “curl 127.0.0.1:8080”. This was not what I was seeing. When I launched the Nginx service, the curl command was successful on the container host where the service was running, but on every other host/node in the swarm cluster, I got a “Failed connect/connection refused”. So why wasn’t it working?

Eventually I traced it to yet another firewall issue on the container hosts/swarm nodes (using Centos 7). It seems that the overlay network needed some ports opened to work as well. These are the ports that I figured out needed to be opened on the firewall of my swarm nodes:

I used the following command on Centos 7 to modify the firewall:

[root@centos-swarm-master ~]# firewall-cmd --zone=public --add-port=7946/tcp --permanent
[root@centos-swarm-master ~]# firewall-cmd --zone=public --add-port=7946/udp --permanent
[root@centos-swarm-master ~]# firewall-cmd --zone=public --add-port=4789/udp --permanent
[root@centos-swarm-master ~]# firewall-cmd --reload

To verify that the changes took place, I used the following command:

[root@centos-swarm-master ~]# firewall-cmd --list-all
public (active)
 target: default
 icmp-block-inversion: no
 interfaces: ens192
 sources:
 services: dhcpv6-client ssh
 ports: 2379/tcp 4789/udp 2377/tcp 7946/udp 7946/tcp 2380/tcp
 protocols:
 masquerade: no
 forward-ports:
 sourceports:
 icmp-blocks:
 rich rules:

The other ports related to Swarm, which is discussed here, and ETCD, which is for vFile (which I haven’t yet blogged about – watch this space). With these ports opened, we have allowed our docker overlay network to communicate between Swarm nodes. Now if I launch a single replica for the Nginx web service and retry the curl test on  different nodes, lets see what happens:

[root@centos-swarm-master ~]# docker service ls
 ID           NAME                 MODE       REPLICAS IMAGE                PORTS
 rxspku5i98cc vFileServerSharedVol replicated 1/1      luomiao/samba-debian *:30000->445/tcp

[root@centos-swarm-master ~]# docker service create --replicas 1 -p 8080:80 --name web nginx
 xvtzr79sb0fdut85yssxd7z1n
 overall progress: 1 out of 1 tasks
 1/1: running [==================================================>]
 verify: Service converged
 
[root@centos-swarm-master ~]# docker service ls
 ID           NAME                 MODE       REPLICAS IMAGE                PORTS
 rxspku5i98cc vFileServerSharedVol replicated 1/1      luomiao/samba-debian *:30000->445/tcp
 xvtzr79sb0fd web                  replicated 1/1      nginx:latest         *:8080->80/tcp

[root@centos-swarm-master ~]# curl 127.0.0.1:8080
 <!DOCTYPE html>
 <html>
 <head>
 <title>Welcome to nginx!</title>
 <style>
  body {
  width: 35em;
  margin: 0 auto;
  font-family: Tahoma, Verdana, Arial, sans-serif;
  }
 </style>
 </head>
 <body>
 <h1>Welcome to nginx!</h1>
 <p>If you see this page, the nginx web server is successfully installed and
 working. Further configuration is required.</p>

<p>For online documentation and support please refer to
 <a href="http://nginx.org/">nginx.org</a>.<br/>
 Commercial support is available at
 <a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
 </body>
 </html>
 [root@centos-swarm-master ~]#

Let’s switch to a worker node, and retry the same test.

[root@centos-swarm-w1 ~]# curl 127.0.0.1:8080
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
 body {
 width: 35em;
 margin: 0 auto;
 font-family: Tahoma, Verdana, Arial, sans-serif;
 }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>
[root@centos-swarm-w1 ~]#

Success! Now that my overlay network is working successfully, I can reach a single instance of a service working on docker swarm from any of the nodes in the cluster.

Exit mobile version