vSphere Distributed Services Engine – Networking Offload and Acceleration Preview
In my earlier overview of vSphere 8 announcements at VMware Explore 2022, I highlight a number of new products and features. One of the most significant announcements is vSphere Distributed Services Engine, formerly known as Project Monterey. As mentioned in the post, this enhancement gives us the ability to offload tasks to a Data Processing Unit (DPU aka SmartNIC). These tasks have historically been done by x86 CPUs on the hypervisor. Now these tasks are offloading to the DPU. And the programmable hardware accelerator on the DPU is also leveraged to boost overall performance.
The first wave of innovation around the vSphere Distributed Services Engine will be to use DPUs to offload network processing through Enhanced Data Path (EDP). EDP is a performant data path model which exists today in NSX-T v3.2. With NSX v4.x, EDP is available in two flavors; EDP-Standard which is interrupt based and EDP-Performance which is polling based. We are recommending EDP-Standard for DPUs at this point in time. When we discuss offloading in this context, we are offloading the networking processing tasks associated with the virtual switch, as well as the networking stack in the virtual machine. The nice thing about utilizing a DPU for these tasks is that it frees up the x86 cores on the hypervisor for more workload orientated tasks. With the new Enhanced Data Path modes in NSX 4.x, we can achieve higher throughput at lower latency with minimal CPU overhead on the hypervisor by offloading to the DPU.
Let’s take a closer look at some of the historical pass-thru options we have had in the past to boost performance. Let’s begin by looking at SR-IOV. SR-IOV has been available on vSphere for some time. SR-IOV, short for single root I/O virtualization, allows a single physical PCIe device such as a NIC to appear to be multiple separate physical PCIe devices. SR-IOV works by introducing the idea of physical functions (PFs) and virtual functions (VFs). Physical functions (PFs) are full-featured PCIe functions; virtual functions (VFs) are more “lightweight” functions. Virtual Machines can attach to a VF on a physical NIC with SR-IOV configured. A simplified overview of SR-IOV is shown below. Note that there is no virtual switch in this picture.
Virtual switches are bypassed when SR-IOV is used. SR-IOV essentially binds the network driver in the VM to a specific VF. This makes it next to impossible to leverage the benefits of virtualization. So, while SR-IOV provides an increase in network performance due to having no x86 CPU overhead for a vSwitch, VMs are effectively “blocked” from using vSphere high-availability and migration features. This makes life-cycle management of such VMs quite challenging.
New Data Path Modes for DPUs
Let’s now turn our attention to the new DPU based data path modes available with NSX 4.x. Once again, this is a preview. Therefore please be aware that some of the screenshots and features which are currently accurate at the time of writing may change when the products become generally available. These new DPU based data path modes are available at a per VM and per vNIC level. The administrator has the ability to fine-tune the offloading and acceleration behaviour. The first of these modes is MUX Mode, which is the default mode. This mode continues to do some processing at the x86 layer but has fewer requirements when compared to UPTv2. The UPTv2 mode completely offloads all processing to the DPU. MUX Mode provides higher flexibility, while UPTV2 provides higher performance. MUX Mode continues to use some x86 CPU on the hypervisor. It has no Guest Memory Reservation, nor does it have a dependency on a specific VMXNET3 driver version. UPTv2 is a complete pass-thru mechanism. It requires Guest Memory Reservation, and has a dependency on a specific VMXNET3 driver version. Both modes provides accelerated networking services for VLAN and overlay networking, as well as offloading TCP Segment Offload (TSO), Checksum and Receive Side Scaling (RSS). In order to configure the new Enhanced Data Path mode with either the MUX Mode or UPTv2 variant, one needs to deploy NSX Manager to configure EDP on the transport node.
UPTv2 requires that the NIC vendor (DPU vendor) publish virtualized device functions (VFs), similar to SR-IOV. Thus, the number of VMs which can be connected depends on the number of published VFs by device vendor. However, UPTv2 gives near SR-IOV performance with almost zero overhead on the hypervisor. And more importantly, it supports core vSphere features such as vMotion for any VMs that use it.
Configuring the new data path modes
So how do you configure network offloading and acceleration to the DPU? Well, as mentioned the configuration does require NSX Manager. If you only want to offload the virtual switch to the DPU, then it is simply a matter of using NSX Manager to enable Enhanced Data Path at the transport layer (on the Transport Node Profile) under the Advanced Configuration settings, as shown below. Remember that we are recommending Enhanced Datapath – Standard for DPUs at present.
It is also necessary to select the correct network offloads compatibility when building the virtual switch on vSphere. The network offloads compatibility should be set to the DPU vendor that you are using. At present, Pensando (from AMD) and NVIDIA BlueField are supported. The virtual switch configuration is completed by choosing physical uplinks from the DPUs.
As previously mentioned in the introduction, offload and acceleration can be configured on a per VM or per vNIC granularity. By default, VMs use MUX Mode for offloading and acceleration. If you wish for one or more virtual machines or vNICs to use UPTv2 mode, then the option to “Use UPT Support” must be set in the Virtual Machine hardware settings for the vNIC, as shown below.
Note once again that this functionality is dependent on vSphere 8.0 and NSX 4.x, as well as having a DPU to offload to obviously. [Update] Note that customers do not need an NSX license for Enhanced Datapath – Standard. Customers who simply want to do distributed switch offload, you do still need to install NSX manager. The NSX manager is made available as a part of vSphere E+ entitlement and will allow you to set the Enhanced Datapath option. Be aware that the Uniform Passthrough feature is not available as part of this entitlement. For customers who want to use UPTv2, or for customers who want to offload overlay networking and the NSX Distributed Firewall to a DPU, these customers will need to acquire appropriate NSX Enterprise Plus licenses.
This is an excellent use case for DPU technologies. I’m sure many of our customers who have latency sensitive and high throughput application requirements will be looking to utilize this functionality very soon. For further information, check out VMware Explore 2022 session CEIB1576US – Project Monterey Behind the Scenes: A Technical Deep Dive. There is also a very good overview video available here: https://youtu.be/Qjbll68I2tk. Expect to see other interesting use-cases from VMware around DPUs over time.