It is quite some time since I looked at deploying NSX-T, VMware’s unified networking platform. The reason for this is that VCF, VMware Cloud Foundation, takes care of the deployment and configuration of NSX-T automatically through the SDDC Manager. However, I wanted to revisit it and do it the hard way, just to re-educate myself on the steps involved. The goal is to have an NSX-T configuration that can be leveraged by vSphere with Tanzu. Since this is rather a lengthy process, I will divide it up into 3 separate posts. The first will focus on the configuration ESXi hosts for NSX-T. The second will focus on configuring the Edge cluster and ensuring that communication can be established between the hosts and Edge nodes. Finally, I will deploy and configure a Tier-0 Gateway and have the environment ready for a vSphere with Tanzu Workload Management deploy.
For completeness sake, here are the product versions that I am using for this configuration.
- vCenter Server 7.0.3c, build 19480866
- VMware ESXi, 7.0.3c, build 19193900
- NSX-T version 18.104.22.168.0.19232597
This post will not focus on all of the discrete deployment steps and requirements, such as distributed switch setup. If you need this level of detail, please refer to the official documentation for deploying NSX-T on vSphere with Tanzu. The following are the steps to deploy NSX-T, assuming all other requirements have been met.
1. Deploy NSX Manager
Begin by deploying the OVA for the NSX Manager. This can now be done from the vSphere client. However, you will first need to download the OVA. These can be retrieved from the VMware Download Site. The NSX-T downloads are here. Note that there are two NSX Manager OVAs. The first NSX Manager is the unified appliance which does not have vCenter integration. The second includes an integrated vCenter plugin to enable deployment and configuration of NSX directly from within the vCenter UI. Either can be used. Once the OVA is downloaded, you can proceed with the install. From the drop down menu at the top left of the vSphere client, select NSX. This will bring you to the following landing page.
Simply click on the Install NSX button to begin the OVA deployment. Select the downloaded NSX Manager OVA. Populate the deployment wizard with the appropriate details, such as IP Addresses, passwords, enabling SSH and storage policies. Once the NSX Manager is ready to be deployed, it should look something like this (this is the unified OVA as per the template name).
2. Compute Managers
Once the appliance is deployed and powered on, you should be able to connect to the NSX Manager UI via a browser, and login as the admin user. Navigate to System > Fabric > Compute Managers in the NSX Manager UI and click on New Compute Manager. Next, provide details about your vCenter server.
If successful, the Compute Manager entry should look similar to the following, with registration and connection status both green:
3. Make highly available NSX Manager
At the moment, the NSX Manager is a single appliance. The recommendation is to scale out to three appliances with a virtual IP address front-end so that the NSX Manager is highly available. Navigate to System > Appliances > NSX Manager in the NSX Manager UI, and then click on Add NSX Appliance. Populate with the appropriate information (name, IP address, gateway, etc).
Once the additional NSX Manager appliances have been deployed, the status is visible within the UI.
When the deployment is complete, the NSX Manager will be highly available. The next step is to configure NSX on the ESXi hosts, but before we do that, there are a number of additional steps that need to be implemented. This is because the node configuration step now requires a host transport node profile before starting the configuration. Therefore the following steps must be done in advance of creating a host transport node profile, and configuring NSX on the ESXi hosts.
4. Create an Uplink Profile for the ESXi hosts
Now we will create an Uplink profile for the ESXi hosts. Navigate to System > Fabric > Profile in the NSX Manager UI. This profile defines which ESXi host uplinks will be used for the overlay network. These uplinks should not be used for anything else on the hosts. The ESXi hosts in this environment are using VLAN 3002 for their host overlay (TEP) network. Thus, the uplink profile looks something like this. Note that the teaming settings here are just labels and not references to the actual uplinks. I have used the labels “uplink5” and “uplink6” to remind me that these are the actual uplinks that I need to map when I create the transport node profile in step 7.
5. Create a Management Overlay Transport Zone
Now navigate to System > Fabric > Transport Zones in the NSX Manager UI. A transport zone will define who can communicate on a particular network. This overlay transport zone will initially just have the ESXi hosts, but later on will include the Edge nodes which will be covered in part 2 of these blog post. Note that the traffic type is set to Overlay and not VLAN.
6. Create an IP Pool for Host Tunnel Endpoints (TEPs)
There are a number of ways in which IP addresses can be allocated to the tunnel endpoints (TEPs) on the ESXi hosts. I am going to use an IP Pool for this. Navigate to Networking > IP Address Pools, under IP Management. Add a subnet of IP address that can be used for the host TEPs, such as those shown below:
7. Create a Transport Node Profile for the ESXi hosts
Using the information created in steps 4, 5 and 6, we can now proceed with the creation of a transport node profile for our ESXi hosts. As you can see, this uses the Transport Zone from step 5, the uplink profile from step 4, and the IP Pool from step 6. At this point, you must also assign the actual uplinks from the ESXi hosts to the uplink mappings, e.g. uplink5 and uplink6.
With the transport node profile created, we can finally proceed with configuring the ESXi hosts as transport nodes.
8. Begin Host Transport Node Configuration
To complete the host configuration as transport nodes, navigate to System > Fabric > Nodes > Host Transport Nodes in the NSX Manager UI. In the “Managed by” section, select the vCenter server that was configured as the Compute Manager back in step 2. Select the cluster, and click on Configure NSX. This will prompt for a Transport Node Profile, as this is where the profile created in step 7 is added:
This will now begin the configuration of NSX on the ESXi hosts.
And if everything is successfully deployed, the following should be the status of the host transport nodes when the configuration is completed.
Note that at this point there are no tunnels shown for the hosts – the status is not available. They will remain like this until a virtual machine is using it. Later on, when we get around to provisioning the vSphere with Tanzu Supervisor cluster, we will begin to see the Tunnels field in the host transport node view populate with information. In other words, this view is normal at this point in the proceedings.
We now have a networking platform that provides us with a policy engine, management plane and control plane, e.g. distributed router/firewall for overlay/tunnels for east-west traffic. In the next post, we will deploy an Edge appliance which will provide us with services for north-south traffic, e.g. firewall, NAT, load balancers, etc.
9. Validate the NSX configuration on the ESXi hosts
To finish this first part of the blog, validate that the NSX configuration on the ESXi hosts is working as expected. Firstly, there should be a number of additional VMkernel adapters. Here is the view from one of the ESXi hosts that has been configured for NSX.
Present in the above listing are both of the overlay interfaces that we configured in the host transport node as well as an additional interface on vmk50, the nsx-hyperbus. This latter interface is the network used by Kubernetes and container creation logic. One final test is to be able to ping between the different hosts on the overlay network. This can be done by logging onto one of the ESXi hosts, and pinging a remote host using the ++netstack=vxlan option to the vmkping commands, e.g.
[root@w4-hs4-i1501:~] vmkping ++netstack=vxlan 192.168.101.1 PING 192.168.101.1 (192.168.101.1): 56 data bytes 64 bytes from 192.168.101.1: icmp_seq=0 ttl=63 time=0.877 ms 64 bytes from 192.168.101.1: icmp_seq=1 ttl=63 time=0.362 ms 64 bytes from 192.168.101.1: icmp_seq=2 ttl=63 time=0.318 ms --- 192.168.101.1 ping statistics --- 3 packets transmitted, 3 packets received, 0% packet loss round-trip min/avg/max = 0.318/0.519/0.877 ms [root@w4-hs4-i1501:~]
Great. Looks like the ESXi hosts have been prepared properly and we can proceed to the next step of building the NSX Edge cluster.