NSX-T Edge Appliance Networking Setup
Again, I’m not going to cover the deployment steps. You can get these from the actual NSX-T documentation here or pop over to Sam’s blog here for the steps. Kudos also to Kevin Barrass for helping us figure out with some of the configuration steps.
Once deployed, the NSX-T Edge will have 4 network adapters. The first of these is used by the management network. The other 3 interfaces (fp-eth0, fp-eth1 and fp-eth2) can then be used for connecting to overlay networks or for routing. We are not going to look at routing just yet. For the moment we will focus on the overlay. Now for the NSX-T Edge and the ESXi hosts to communicate on the overlay, the NSX-T Edge needs to have a connection to the same VLAN/network that the ESXi hosts are using for the overlay. So you will need to create a portgroup on a VSS/VDS which has an uplink that can reach this VLAN/network. This is what my NSX-T Edge configuration looked like before I begun to add it to my NSX fabric. If you do not have a VM portgroup on a VSS/VDS that can reach the overlay network setup on the hosts, you will have to create one and connect it to one of the interfaces on the NSX-T Edge as shown below.
Do not worry about the router connection for now. My router will eventually be on my management network, as this is how I will be connecting my internal networks to the outside world (which I will try to cover in a later post if I can).
Adding the NSX-T Edge to the NSX Fabric
The easiest way to do this is to run the following 2 commands, one on the NSX-T Manager and one on the NSX-T Edge. The first step is to get the “thumbprint” and the second is to join the Edge to the fabric. Login to the manager as admin:
nsx-manager> get certificate api thumbprint
On the NSX-T edge, login as admin and run this command:
nsx-edge> join management-plane nsx-manager username admin thumbprint \ 75e7dfb8414496b984ff57cdf549ec889a4ca8ae9fbc7aaf206dc1679d2a1949
Password for API user:********
Node successfully registered as Fabric Node: fc943c22-3e29-11e8-98a4-005056bcba93
The NSX-T Edge should now be visible in the NSX-T Manager UI, under Fabric: Nodes, then via the Edges view.
Create an Edge Profile
Just like we did with the hosts, the Edge will also need it owns profile. This is different to the host profile. The first is that there is no need to add a VLAN ID since the traffic is already tagged at the vSwitch layer. So we can set the VLAN to 0. The second is the MTU size. This is set to 1500 rather than 1600. This is what my Edge Profile looks like:
Making the Edge a Transport Node
Again, just like we did with the hosts, the Edge will now be configured as a Transport Node, and added to the same Transport Zone as the hosts. Go to Fabric: Nodes, click on Transport Nodes and click +ADD. Below is the General view, where we simply provide the name, the node (which is our new Edge) and move the Transport Zone (same one used by our hosts) from Available to Selected.
Next, click on the N-VDS part to fill in the details about the hostswitch. Here we add the Edge Switch Name which is the same as the hostswitch. The “overlay-switch” is, in this example, the name of the hostswitch that was configured for the hosts when making them into Transport Nodes. It just now means that the hosts and the edge are on the same tunnel.
The uplink profile is the one we just created earlier. The IP assignment IP Pool, “hosts-teps-pool”, is the same one I used when making the ESXi hosts Transport Nodes.
Finally, select a network adapter on the edge to connect to the overlay network. fp-eth1 below is network adapter 2 on my Edge as seen from vSphere, and this is the one I connected to the portgroup (nsx-t-edge-connect) that can reach the network used by the hosts for the overlay. Here are the details that I used.
When the above is saved, the Edge should now be visible as a transport node, alongside the hosts.
Verify Edge to Host overlay connectivity
The Edge should be assigned the next available IP address from the IP pool above. My hosts were assigned 192.168.190.1-4 respectively, so my Edge should have been assigned 192.168.190.5. A way of checking is to SSH onto your Edge as the admin user, and run the following commands. First, get the VRF id of the TUNNEL, and use that to get the list of interfaces. This should verify that the Edge tunnel/overlay connection is indeed the next IP in the IP pool range, in this case 192.168.190.5.
nsx-edge> get logical-routers Logical Router UUID VRF LR-ID Name Type Ports 736a80e3-23f6-5a2d-81d6-bbefb2786666 0 0 TUNNEL 3 nsx-edge> vrf 0 nsx-edge(vrf)> get interfaces Logical Router UUID VRF LR-ID Name Type 736a80e3-23f6-5a2d-81d6-bbefb2786666 0 0 TUNNEL interfaces interface : 9fd3c667-32db-5921-aaad-7a88c80b5e9f ifuid : 262 mode : blackhole interface : 005c7fff-29b3-5355-b4a2-89b4a0e9216b ifuid : 309 name : mode : lif IP/Mask : 192.168.190.5/24 MAC : 00:50:56:bc:58:a3 LS port : c9470dfe-cbac-581e-9b64-67dbfab6c056 urpf-mode : NONE admin : up MTU : 1600 interface : f322c6ca-4298-568b-81c7-a006ba6e6c88 ifuid : 261 mode : cpu nsx-edge(vrf)>
The final test is to verify that we can ping the Edge interface from the ESXi hosts, and vice-versa. From the Edge session above, you can use a normal ping command, but from the ESXi hosts, you need to specify that this is the special “Geneve” tunnel/overlay network, so you will need to use the ++netstack=vxlan option, e.g.:
vmkping ++netstack=vxlan 192.168.190.5
Check out the previous post on overlays if you need more guidance on this step. If that succeeds, congratulations. You just joined your edge to the same overlay as your hosts.
Create an Edge Cluster
In order to get our Edge to do anything useful, such as facilitate DHCP, it needs to be part of an Edge cluster. Even though we only have one Edge, this is still a necessary step. Navigate to Fabric: Nodes, select Edge Cluster, then +ADD, and simply add your Edge to the Edge Cluster. The Transport Node type needs to be changed from Physical Hosts to Virtual Machines for it to become visible. Then move it from Available to Selected, like so:
Set up DHCP
Now we are ready to set up DHCP. The are 5 steps to getting DHCP deployed.
- Create a DHCP Server Profile
- Create a DHCP Server
- Add an IP Pool to the DHCP Server
- Create a Logical Switch
- Attach the DHCP Server to the Logical Switch
Let’s get going.
Create a DHCP Server Profile
In the NSX Manager UI, navigate to DDI: DHCP and select Server Profiles. Click on +ADD and give the profile a name. If you only have a single Edge Cluster, this will be automatically populated with the name of the Edge Cluster created previously. Select which Edge Transport Node to use for this profile. Again, in my setup there is only one. Finally, save the profile.
Create a DHCP Server
Navigate to DDI: DHCP and select Servers, then click on +ADD to add a DHCP Server. All that is required for the DHCP Server is a name and an IP address. I have given it an IP address of 192.168.191.2 as I plan to use 192.168.191.1 as my gateway later for getting to the outside world. I haven’t populated any other fields. Now save the DHCP Server.
Add an IP Pool to the DHCP Server
This part was a little confusing. You have to click on the name of the DHCP server, and on the right hand side which has the DHCP Server Overview details, there is an IP Pools section. Expand this section, and click on +ADD to add an IP range for your DHCP server. I have chosen the range 192.168.191.100-192.168.191.200. I have also set a default gateway of 192.168.191.1. Leave the rest of the settings at the defaults. Save the IP Pool.
Create a Logical Switch
In this step, we are creating a logical switch. This is different to the hostswitch/N-DVS that we had to create earlier for the Transport Nodes. The logical switch will appear as a portgroup in vSphere, and we will be able to connect our Virtual Machines network interfaces to it. VMs connected to the same logical switch will be able to communicate via the overlay network. Navigate to Switching: Switches and click on +ADD to create a logical switch. You simply need to provide a name (which will also be the portgroup name) and the Transport Zone (the same Overlay transport zone we have been using all along). There is VLAN requirement, and Admin Status (Up) and Replication Mode (Hierarchical) can be left at the default. Save the Logical Switch.
Attach the DHCP Server to the Logical Switch
Our final step is to attach the DHCP Server to the Logical Switch. If our DHCP server is configured correctly and attached to the logical switch, it should be able to provide our VMs that are attached to this logical switch/portgroup with an IP address. Simply select the Logical Switch (check the box against the Logical Switch in the Switching: Switches view, then select the Actions drop-down, and click on Attach to a DHCP Server.
From the pop-up, simply select the DHCP Server that we created earlier and click save.
This step could also have been done from the DHCP server, where its actions list allows it to be attached to a Logical Switch. It doesn’t really matter which method you choose.
Verifying Switch and DHCP functionality
The first thing we can check is the DHCP server port on the Logical Switch. Click on the Logical Switch, and in the Overview section on the right hand side, you should see the port listed if you select the Related: Ports view.
Next, we can go to our vSphere environment, and deploy a VM that will use the logical switch/portgroup and see if it gets allocated an IP address. I chose a Windows VM, and simply picked the logical switch/portgroup as follows. Note the name of the portgroup is the same as the logical switch.
Now if I open a console to that VM, I can check the IP addresses, etc. In this VM, I can see that this VM has indeed been assigned an IP address from the DHCP range configured earlier.
Excellent. Our Edge is doing exactly what we wanted it to do. And I can also ping the DHCP server from this VM:
Now we have not yet set up any gateway, so there is no point in pinging that just yet. The next thing we can do is to verify that I can also do VM to VM communication. Therefore I configured another VM to use the logical switch/portgroup, and it also get its IP address from our DHCP server.
And as a final check, can we now ping our other VM from this VM on the tunnel?
Success! These VMs are also visible on the Logical Switch back in NSX Manager. Navigate to Switching: Switches, click on the overlay logical switch, then in the Related section, drop down and select Ports. Here you will see the VMs that are connected to the logical switch/portgroup, as well as the DHCP server port.
Great. Now we have an NSX-T Edge, participating as a Transport Node in our Transport Zone Overlay/Tunnel. We then configured our NSX-T Edge to provide DHCP functionality, attached it to a Logical Switch, which in turn allowed this service to be consumed by our VMs. Very nice.
Stay tuned while I figure out my next steps, which is basically to get this overlay network/tunnel routed to the outside world.