NSX-T Edge T0 Router: Static Route + SNAT instead of BGP
If you been following my adventures of deploying Enterprise PKS 1.5 on VMware Cloud Foundation (VCF) 3.9, you will be aware that I spent a considerable amount of time establishing Border Gateway Protocol (BGP) peering between my NSX-T Edge T0 Logical Router and my physical Upstream Router as documented in this post. This allows them to exchange route information, so that when one of my internal overlay networks needs to communicate externally, it can do so. However, I am in the fortunate position where I can access my Upstream Router and make any necessary BGP configuration changes to allow it to establish a connection to my NSX-T Edge T0. Not everybody may be in a position to do this, and there may be many cases where the network administrator may not allow you to establish a BGP connection from your NSX-T Edge to the upstream router, especially if your setup is a Proof-Of-Concept or a test/demo lab environment. Is there another way to allow the internal overlay networks to route externally? Yes, there is. You can configure a static route and a source NAT rule on the NSX-T T0 to achieve this. In this post, I am going to assume that VLAN 51 has external access, but VLAN 51 is managed on a core upstream router to which I have no access. I am therefore going to create a new NSX-T Edge T0 Router port which is on this VLAN 51, and add a static route for all internal overlay networks to route via this uplink to gain external access. I will then add a simple Source NAT (SNAT) rule, which NATs all the source traffic to the IP address of the external router port.
Note: Most of the instructions to configure an NSX-T Edge, end-to-end, are already covered in the my Getting started with VCF Part 7 – NSX-T Edge post references above. I’m not going to go into as much detail here as I did on that post. The steps shared here work off of the assumption that you have read that post and are familiar with NSX-T Edge configurations. If not, please use that post for detailed step, and converge them with the steps in this post.
Step 1 – Create an uplink profile on VLAN 51
The first step is to create an uplink profile as we are going to use one of the uplinks on the NSX-T Edge appliance to connect to VLAN 51. The assumption here is that the network to which the physical uplink on the Edge appliance is attached is a trunk port which include VLAN 51. Therefore set the Transport VLAN to 51. Note that I am using the last uplink on the NSX-T Edge, Fp-eth2. Since this is not an overlay/tunnel, you do not need a larger MTU size so 1500 MTU is fine.
Step 2 – Create a Transport Zone for type VLAN
Next, we need to build a Transport Zone, which can be added to the Edge Transport Node. You need to provide a unique N-VDS Name (pick something that is easily recognizable) and choose VLAN rather than Overlay for Traffic Type.
Step 3 – Configure NSX-T Edge Transport Node with new Transport Zone
Add the new Transport Zone to the Edge Transport Nodes in the General tab. In the N-VDS tab, add a new N-VDS. Set the N-VDS configuration to choose the correct Edge Switch Name that you created in the Transport Zone in step 2, choose the uplink profile that you created in step 1 and finally choose the virtual NIC (Fp-eth2) that you want to configure. This is why I set the Active Uplink to Fp-eth2 in step 1, as it makes it easier to map here.
 |
 |
Your Edge might now look something like this, although some of the field may be different if this is a brand new Edge deployment. Just make sure the Configuration field shows Success.
Step 4 – Create an External T0 Router Port (Logical Switch)
What we are in effect doing in this step is building a Logical Switch. This will act as a Logical Router Port. This is the port which will connect the T0 Logical Router to VLAN 51, so it also need to be on VLAN 51.
Step 5 – Attach VLAN 51 External T0 Router Port to the T0 Logical Router
The assumption is that the logical router already exists. If you need guidance on how to create one, see the previous post linked in the introduction. Set the MTU to 1500, and select the Logical Switch (T0 Logical Router Port) created in step 4. In the Subnets section, provide this port with an IP address on the external network.
Step 6 – Validate that you can reach the external network
At this point, from the NSX-T Edge Service Router, you should be able to ping IP addresses on VLAN 51, not just the T0 logical Router port, but also other addresses on the network. SSH onto the NSX-T Edge appliance, and run the following commands. Be aware that this is an existing environment, which also has a T1 switch configured. The number of logical routers displayed in your environment may be different. However, the tier 0 service router should exist and that is the one you will select below.
w01-edge-01> get logical-router Logical Router UUID VRF LR-ID Name Type Ports 736a80e3-23f6-5a2d-81d6-bbefb2786666 0 0 TUNNEL 3 267cf518-f1fd-4b13-9008-5e0bf2662a96 5 3074 SR-T0 SERVICE_ROUTER_TIER0 6 14b5a772-fa42-45b3-b5f5-695584a9f5b6 6 3076 SR-T1 SERVICE_ROUTER_TIER1 5 802116e0-2639-46ad-90b3-33edafadd5d8 7 3073 DR-T0 DISTRIBUTED_ROUTER_TIER0 4 07557296-2a54-4e76-9146-9b9da9600d11 9 3075 DR-T1 DISTRIBUTED_ROUTER_TIER1 5 w01-edge-01> vrf 5 w01-edge-01(tier0_sr)> ping 10.27.51.253 PING 10.27.51.253 (10.27.51.253): 56 data bytes 64 bytes from 10.27.51.253: icmp_seq=0 ttl=64 time=1.184 ms 64 bytes from 10.27.51.253: icmp_seq=1 ttl=64 time=0.748 ms ^C w01-edge-01(tier0_sr)> --- 10.27.51.253 ping statistics --- 3 packets transmitted, 2 packets received, 33.3% packet loss round-trip min/avg/max/stddev = 0.748/0.966/1.184/0.218 ms w01-edge-01(tier0_sr)> ping 10.27.51.254 PING 10.27.51.254 (10.27.51.254): 56 data bytes 64 bytes from 10.27.51.254: icmp_seq=0 ttl=255 time=3.439 ms 64 bytes from 10.27.51.254: icmp_seq=1 ttl=255 time=0.933 ms ^C --- 10.27.51.254 ping statistics --- 3 packets transmitted, 2 packets received, 33.3% packet loss round-trip min/avg/max/stddev = 0.933/2.186/3.439/1.253 ms w01-edge-01(tier0_sr)>
Step 7 – Add a Static Route to the T0
So we are not going to use BGP here to learn about external routes. Instead, we will use a static route. I still want all of my overlay networks to be able to route externally to the 10.27.51.0/24 network. Navigate to Advanced Networking and Security > Routers > Routers and click on the Tier 0 router. From the Routing dropdown, select Static Routes and add the appropriate entries. Add a static route to the 10.27.51.0/24 for all internal networks. The Next Hop is the gateway IP address on that network.
Note that this static route may not be good enough for your purposes. You may want to route to many external networks. An example of this would be the ability for Kubernetes nodes to reach one or more external container repositories/hubs for container images. You can make this wide open to all external routes by adding an IP/mask of 0.0.0.0/0.
Step 8 – Add a Source NAT (SNAT) rule
At this point, an internal overlay networks will be able to route out to VLAN 51, but anything on that VLAN will not be able to reply/respond to it as it won’t know how to route the reply. In other words, something might receive a ping from the 20.0.0.0/24 network, but there is no route on the physical upstream router to that network, so how can it respond? This is where BGP is much simpler, as it automatically learns about routes from its peers, e.g. an NSX-T T0 Logical Router. Therefore, in order to overcome this issue with using static routes, we can add a source NAT for all our overlay networks. This means that all overlay traffic will be translated to an IP address on VLAN 51. Here I am using our router port IP address as the SNAT Translated IP address. With this rule in place, we can now ping out from virtual machines on our overlay networks to VLAN 51, and the destination of the ping responds to 10.27.51.253, which in turn knows how to route the response back to the origination overlay network.
Step 9 – Pinging the overlay network from VLAN 51
At this point, we have configured a static route and a SNAT so that we can successfully communicate externally from our overlay network, but how do we ping from an external (VLAN 51) device to the overlay networks. These hosts/VMs (for example) do not know how to reach the overlay networks. To overcome this, you need more static routes, but this time on the hosts/VMs that wish to ping the overlay network.
Here is an example of how to add static route to reach the overlay networks (147.70.10.0/24, 147.70.20.0/24) from a Windows VM on VLAN 51.
Use the route print command to display the routing table. You can see the routes using the T0 Uplink Port Ip address as the gateway. This is also a SNAT address of course.
A similar command is run on Linux VMs. This is in fact the SDDC Manager for my VCF deployment.
Summary
You can indeed use a static route with SNAT to achieve routing in and out of your overlay networks with an NSX-T Edge Tier 0, but you will have to add additional static routes to each of the hosts/VMs that wish to communicate with devices reading on the overlay networks, which can be a bit of a PITA. This is why BGP is a more elegant solution, but as mentioned at the beginning of the post, it may not always be possible to access the upstream router to configure it. This method of static route with a SNAT provides a work-around for those environments (POCs, labs) that may not be in a position to setup BGP.
