I found it easier, not being a network person, to visualize my network layout as follows. Hope these make sense. This first diagram shows where we are once the overlay network and the DHCP servers have been configured.
And this next diagram shows us our desired outcome, with the VM networks connected to the logical switches able to route to the outside world through out T0 and T1 logical routers. Note that the route from the T0 Logical Router and my Physical Router is via the NSX-T Edge. We have to use another one of the ports on the NSX-T Edge for this external connection. We will see how to do this shortly.
Let’s talk about the 12 steps involved.
- Create a new Transport Zone of Type VLAN (not of type Overlay)
- Modify the Edge Transport Node (created when Edge was deployed) and add the new VLAN Transport Zone
- Create a new Logical Switch using the new VLAN Transport Zone
- Create a T0 Logical Router
- Connect the VLAN Logical Switch to the T0 Logical Router Port – requires an IP address that can connect to the physical switch VLAN gateway IP
- Create a T1 logical router
- Connect the VM logical switches (191, 192, 193) to the T1 router
- Connect the T0 to the T1
- Ping the external IP of your Logical Router from your VMs/Advertise the routes
- Setup Route Distribution
- Setup BGP on the T0 – pair your T0 with your physical router
- Setup BGP on your physical router – pair your physical router with your T0
OK. Now lets look at those steps in greater details.
1. Create a new VLAN Transport Zone
This step is quite straight-forward. Navigate to Fabric > Transport Zone in the NSX Manager, and click + to add a new TZ. We already have an overlay TZ, but now we need a VLAN overlay to reach outside. Give the new TZ zone a name and ensure you select VLAN as the Traffic Type.
2. Add the Edge Transport Node to the new VLAN Transport Zone
We need to associate the new Transport Zone with the Edge Transport Node. There are a few steps involved here. First you will need to navigate to Fabric > Nodes, select Transport Nodes, select the Edge TN, click on Actions and choose “Add to Transport Zone”. In the General tab, select the VLAN Transport Zone created earlier and move it from Available to Selected.
Next, click on the N-VDS tab. You will need to create a new hostswitch for this Transport Zone, alongside the existing hostswitch that was already created for the overlay. Click on +ADD N-VDS. Give it a name, select the same uplink profile created for the overlay hostswitch, and select the correct Virtual NIC on the Edge. In my case, the first available uplink on the Edge (not the one connected to the management network, but the next one) is the one I am going to use for my connection to the outside world. This is fp-eth0. I used fp-eth1 for the overlay network connection previously. fp-eth2 is unused on my Edge.
Click Save. Your Edge Transport Node will now be in two transport zones, the overlay created earlier and the new VLAN TZ just added.
3. Create a new Logical Switch with the new VLAN Transport Zone
We must now create a new Logical Switch. This Logical Switch will later be used to connect one of our Logical Router ports to the outside world (via the Edge uplink). Navigate to Switching > Switches. Click on + to add a new Logical Switch. Give it a name, and ensure that the Transport Zone is set to the one that can handle VLAN traffic that we created earlier. Set to VLAN ID to 0 – any VLAN tagging will be done outside of NSX-T.
4. Create a T0 Logical Router
Now we come to the step where we start to create the routing infrastructure. Navigate to Routing > Routers, and click on +. Then select “Tier 0 Router“. Give it a name, select the Edge cluster (created when we set up the DHCP servers) and since I only have a single member in the Edge cluster, I selected Active-Standby and provided the name of my Edge Transport Node, edge-TN.
5. Connect the VLAN Logical Switch to the T0 Logical Router via New Router Port
The next step is to add a router port to the T0 Logical Router which connects to the Logical Switch that in turn connects our router to the outside world. Select the T0-LR just created, and when the detail populate on the right hand side of the screen, select Configuration > Router Ports, then click on +ADD. Give it a name, leave the type as Uplink, select the logical switch which can carry VLAN traffic that we created earlier, and finally give this port an IP address on the external network to which we are connecting (the other end of this connection will be the VLAN gateway IP address on the switch, which will become obvious when we configure BGP later on). This is the same network that the gateway for the physical switch’s port resides. This will add a new router port to the T0 logical router.
6. Create a T1 Logical Router
Now we will build a T1 Logical Router. This will be used to connect our Logical Switches. Navigate back to Routing > Routers, and click on +. Then select “Tier 1 Router“. Give it a name, and select the Edge Cluster from the drop-down. You can of course connect to the T0 Logical Router at this time, but I chose to do it later, so I have left this blank.
7. Connect the VM Logical Switches to the T1 Logical Router
Staying in Routing > Routers, and with the T1 Logical Router selected, select Configuration > Router Ports. We will now add the Logical Switches to this T1 Logical Router. I had created 3 Logical Switches with 3 different DHCP ranges, so I will add all 3 as I want all of my VMs to be able to reach the outside world. Use the +ADD to add a new port, give the port a name, select the Logical Switch, and then give this port an IP address on your VM network. This will become the default gateway connection for my VM logical network (VMs connected to this logical switch). I used .1 for the router port that my logical switch is added to and .2 for my DHCP server on these networks. My DHCP range can then be every other IP on the network.
8. Connect the T0 to the T1
At this stage, the T0 LR is connected to the outside world, and the T1 LR is connected to our Logical Switches/VM networks. Now we can connect the T0 Logical Router to the T1 Logical Router. This can be done from either the T0 or the T1. From the T1, click on the Actions, simply select the correct “Connect to Tier-0 Router” option from the drop-down. At this point, every thing is connected.
After connecting the T0 to the T1, lets take a look at the port configurations on both. First, the T0-LR ports. We can see the connection to the L1, as well as the uplink connection to the outside world.
And now the T1:
Here we can see the connection to the T0 logical router as well as the 3 down-link connections to my logical switches, where my VMs are attached. Now you could have approached this differently of course, using multiple edges and thus multiple T1 Logical Routers as a way of balancing load across Edges (and even ESXi hosts if you used affinity/anti-affinity rules with DRS). However I’ve tried to keep things simple so I put all my logical switches on the same Edge and the same T1 Logical Router.
9. First ping test/Advertise the routes
At this point, my VMs will still could not reach the router port IP address. To get this to work, I needed to enable route advertisement on the Tier 1 router. Select the T1 LR, and under Routing, select Route Advertisement. Click on enable and set Advertise All NSX Connected Routes to Yes. Now the ping test from my VMs is working to the uplink port created on the T0. Note that in certain cases, you may not want to have all of the other routes advertised, especially if you are using some internal IP address ranges which are used elsewhere on your network.
However, at this point, I still cannot ping the outside world. To do that, the simplest way is to use BGP, the Border Gateway Protocol, and allow the T0 to learn about the physical router, and vice-versa. Let’s do that next.
10. Setup Route Distribution and ping the outside world
The next step is to enable Route Distribution and add the appropriate criteria. According to the documentation, by default, routers share routes with other routers running the same protocol. In a multi-protocol environment, you must configure route redistribution for cross-protocol route sharing. If I understood correctly, this feature avoids us having to set up BGP across all hosts for physical infrastructure connectivity. Anyway, this is a necessary step if you want to be able to ping the outside world from a VM residing on an NSX-T logical switch, and vice-versa.
To configure it, navigate to the T0 Logical Router, click on the T0, and under Routing, select Route Distribution. The first step is to enable and save it:
Next, you need to add the appropriate criteria. Click on the +ADD option to do this. Give the criteria a name and select the appropriate sources. For this very simple setup, the only Source that I needed to select to get pings from my NSX-T logical switches to other VMs on the outside world was as the NSX Static setting.
11. Configure BGP on T0 Logical Router
Border Gateway Protocol (BGP) is a protocol designed to exchange routing information among autonomous systems (AS). To get this to work, BGP needs to be configured on both the T0 Logical Router in NSX-T as well as the physical router. Let’s look at how to do that now on the T0 Logical Router.
Step 1 is to simply enable BGP and assign the NSX-T T0 Logical Router with an AS (autonomous system) number. Navigate to your T0 Logical Router, select Routing, then BGP. In this case, I used 999 as my local AS.
Save this. Then under the neighbors section, click on +ADD to add details of your physical router, such as IP address and its Remote AS. My physical router IP address (default gateway for the VLAN to which my Edge VLAN uplink is connected) is 220.127.116.11. The AS in this case is 5500. In the next section, I will show how to configure this.
With this information saved, you now need to turn your attention to the physical router.
12. Configure BGP on Physical Router
There are a number of steps required if BGP is not already enabled on your switch. Here are some of the commands needed:
- If your VLAN interface does not have a default gateway/ip address associated with it, then this needs to be done first.
- Next, define your router’s autonomous system number, e.g. – router bgp 5500
- Track BGP neighbor updates, e.g. – bgp log-neighbor-changes
- Set a Router ID, e.g. – bgp router-id <router-id>
- Set the times to match the NSX-T T0 Logical Router, e.g. – timers bgp 60 180
- Add the T0 as a neighbor using its AS, e.g. – neighbor 18.104.22.168 remote-as 999
You should refer to your switch vendor documentation for the exact steps.
There are now a bunch of commands, both on the NSX-T edge, and on the physical switch to look at the BGP neighbor information and verify it is working. One example from the physical switch is show ip bgp, and in this case I can see my 3 logical networks all discovered (192.168.191.0/24, 192.0/24 and 193.0/24). There are also other neighbors, but these are for different networks. This is because there is another BGP configuration for a completely different T0 Logical Router from another deployment.
Similar, from the NSX-T edge, from the T0 LR, you can query the BGP neighbors. In this case I used the command get bgp neighbor after selecting the tier0 service router VRF. To get to this output, logon to the NSX-T Edge as an admin user, run the command get logical-routers, note the VRF for the tier0 service route, and type vrf , then run the command:
At this point, after saving the criteria, you should now be able to reach the outside world, and successfully ping IP addresses on the external network from your virtual machines residing on NSX-T logical switches. And you should also be able to ping your NSX-T logical switch based virtual machines from externally on the network. Remember that if you are pinging an NSX-T networked VM from an external multi-homed VM that has a default gateway that is not pointing to the same network as your NSX-T T0 Logical Router, then you will have to specify the correct adapter in the ping command.
This was an interesting learning exercise for me, just to see the different moving parts of NSX-T. Again, this configuration is very limited by my own physical network, and I strongly suspect that this is probably not something you are going to see in real-life. However, I do think it is useful in showing some of the routing features and capabilities of NSX-T.
Much kudos to my colleague Paudie, and our good friend over at DELL-EMC, Keith Lee, for guidance through some of the BGP stuff. Keith said he is about to start a blog series on getting PKS working with NSX-T, so probably a good time to start following him on twitter.