For as long as we’ve been doing Network Virtualization, there has been debate about how best to encapsulate the data. As we pointed out in an earlier post, it’s entirely reasonable for multiple encapsulations (e.g. VXLAN and STT) to co-exist in a single network. With the recent publication of “Geneve”, a new proposed encapsulation co-authored by VMware, Microsoft, Red Hat and Intel, we thought it would be helpful to clarify a few points regarding encapsulation for network virtualization. First, with all the investment made by us and our partners in developing support for VXLAN (described here), we very much intend to continue supporting VXLAN — indeed, we’ll be enhancing our VXLAN capabilities. Second, we want to explain why we believe Geneve is a necessary and useful addition to the network virtualization landscape.
As we have often stated, and as shown above, a tunnel with its associated encapsulation is analogous to a cable. Based on the capabilities of the destination, the source endpoint can encapsulate the inner frames within the appropriate tunnel header and send them out as IP packets. Multiple tunnel types could be supported over the same IP infrastructure – e.g. Geneve between two hypervisors, VXLAN between another set of hypervisors, STT between a third set and finally, VXLAN from the hypervisor to the gateway. Unlike hardware cables, software permits the configuration of this capability to be performed dynamically and automatically. In the case of NSX, the network virtualization controller can ensure that each pair of communicating endpoints uses the appropriate encapsulation. When the endpoints are implemented in software, it is easy to upgrade or enhance the “cabling” technology. This is something that only the two endpoints care about. The upgraded/modified cables may now facilitate the transport of richer information (e.g., to carry additional parameters for the endpoint to use in processing the information). The other endpoints using existing cable types can work without any changes. This is exactly how we see the deployment of Geneve — it will not impact existing VXLAN deployments.
This last point warrants some further comment. Clearly there is a lot of VXLAN-capable hardware out in the marketplace now, with more to come, and this hardware will continue to exist in data centers for many years. We’ve invested a lot of effort working with partners — and continue to invest — to enable the creation of virtual networks that span the physical and virtual worlds. In every case, we’ve used VXLAN as the “cable” to connect third party switches to hypervisors, because that’s what the hardware supports today. VXLAN is a great “version 1” protocol for network virtualization, having received support from all the major switch hardware vendors. We’ll continue to use it as long as there is hardware out there supporting VXLAN, which we expect to be a long time. As noted above, it’s easy for a network virtualization controller to support multiple encapsulations based on the capabilities of specific endpoints, and to have multiple encapsulations used at the same time in the same network — just as you might find a mix of twisted pair and optical fiber in the cabling of a data center. This doesn’t change with the introduction of a new encapsulation.
You might ask then, if we’re going to keep on using VXLAN to talk to current generation hardware, why introduce Geneve? Primarily, we wanted to combine the best of the current network virtualization encapsulations (VXLAN, NVGRE, and STT) into a single protocol that could do all the things that those protocols do, and more. We drew on our experience from many years of network virtualization development, and one thing that really stood out was the need for an extensible protocol. (See this post for a nice discussion of how important this is.) For example, in the case of STT, we included a 64-bit metadata field to pass network virtualization parameters. One example of metadata is the virtual network identifier (VNI), but there are lots of other examples. We continually found new uses for this metadata field as the product evolved and we added new features like private VLANs, distributed routing, etc. In fact, we began to realize that at some point, 64 bits, which initially seemed like a lot, might not be enough. Hence the decision to encode metadata in an extensible TLV structure within Geneve. The TLV structure also makes it easy for hardware to skip over any metadata types that it cannot or does not need to understand. Thus it becomes possible to independently evolve the capabilities of both software and hardware endpoints as new requirements emerge.
What about switches in the underlay? The tunnel encapsulation has no requirements from the underlay except for IP transport. This continues to be the case with Geneve. Also, like VXLAN, the switches in the underlay can continue to distribute traffic across links using ECMP based on the outer packet header fields (Src IP, Dest IP, Protocol [UDP], Src Port, Dest Port) which are identical for both encapsulations. The underlay can continue to support both Geneve and VXLAN with no changes. Tools for monitoring such as wireshark can be easily enhanced to recognize and parse Geneve information as well.
Another place where encapsulation affects hardware is the NIC. We have been working for several months with our hardware ecosystem to ensure that they will support both Geneve and VXLAN in their next generation silicon. This includes NICs that provide stateless offloads like CSO (Checksum Offload), LSO (Large Segment Offload) and LRO (Large Receive Offload). As there will be a mix of older and newer NICs in real deployments for some time, this will be another case where supporting multiple encapsulations in the same network may be necessary – just as we do today with VXLAN and STT.
The authors of the current Geneve draft include Intel, Microsoft, Red Hat and VMware. The aim from the beginning was to make this a multivendor effort (similar to VXLAN) with broad support from partners and the ecosystem. The hardware ecosystem can benefit from an industry consensus on a single encapsulation for the long term, removing the need to support multiple encapsulations in their silicon. This is why a unified but flexible encapsulation was desired. It is true that, in the short term, we’ve increased the number of encapsulations by one. But the new encapsulation format provides the required flexibility and creates the possibility that we won’t have to keep on defining new encapsulations every few years.
In summary, a flexible and extensible tunnel format like Geneve enables rich capabilities for network virtualization both for current and future use cases. VXLAN will continue to be deployed as one type of tunnel encapsulation to enable network virtualization. VMware will continue to support VXLAN both separately and when coexisting with Geneve. Geneve will be used for more advanced and rich use cases – including, probably, some that we have not yet envisaged. That, we believe, will be the value of a unified and extensible encapsulation.
This post was co-authored by Bruce Davie and T. Sridhar.
