VXLANs and the Cloud Infrastructure Suite…

Phew, nothing like spending time at VMworld, which has become the new mecca for IT professionals everywhere – thanks again to everyone for the support and encouragement, egging us to keep on truckin’!

In my last post, I talked about the need to have logical networks, edges and trust zones on a per tenant basis, and the need to map these onto provider networks that are increasingly becoming fast, fat and flat. At VMworld 2011, we announced some key advancements that make these concepts a reality; VMware and partner booths demonstrated a variety of Software Defined Networking solutions, leveraging VXLANs, vCloud Director, vShield and vSphere/vCenter, all part of the emerging Cloud Infrastructure Suite – we are getting closer to what Paul Maritz refers to as the “invisible infrastructure”…

Let’s talk through some of the advancements in network & security virtualization, realizing elastic Virtual Data Centers (VDCs), and VMware’s ecosystem framework…

VXLAN – the protocol

One of the bigger announcements made at VMworld, was the VXLAN initiative. See Steve’s post on the subject, and this writeup from Mallik Mahalingham, Principal Engineer at VMware, who has spearheaded this effort at VMware.

From a wire-protocol perspective, VXLAN is essentially a MAC-in-UDP frame format, including a 24 bit segment Id. Effectively UDP gets you to the right ESX host/controller, the segment Id gets you to the respective Org/Tenant, and the “inner” MAC gets you to the right vNIC/VM in the Org. Additionally, tenant broadcasts are converted to IP multicasts (Protocol Independent Multicast – PIM).

Here are a couple of posts from networking blogger Ivan Pepelnjak, one on VXLAN versus OTV/LISP, and one on vCloud Networking. Worth a read.

VMware has been collaborating with Arista, Broadcom, Brocade, Cisco, Emulex, Intel and several industry players to work towards standardizing this protocol. With the broad collaboration, expect to see VXLANs become the currency for multi-tenant data center networking (enabling multi-tenant data center fabrics), with NIC vendors providing native offload/acceleration of VXLAN frames, Top of Rack and switch/router vendors providing fast, fat and flat implementations of the protocol, and vendors building VXLAN-VLAN gateways to enable high performance, mixed environments.

What’s nice about VXLANs is that it leverages existing infrastructure, yet can take advantage of advances in data center fabric and server NIC technologies as these become available. However, the really big deal here is that the broad (virtualization, system/NIC, and networking vendors) convergence on the frame format, enables the industry to move on, and focus on innovation in the data center fabric, within the virtualization layer, and in delivering capabilities to cloud tenants/orgs.


Building logical networks on the VXLAN foundation

So, on the infrastructure provider side of the equation, VXLANs provide us the capability to realize isolated, multi-tenant broadcast domains across data center fabrics. Let’s talk about how VMware and partners leverage this capability in emerging provider networks to provide elastic, logical networks to tenant/org VDCs.


The Virtual Distributed Switch abstracts the data center fabric and provides a sea of ports. vCloud Director (VCD) creates an Org Virtual Data Center (VDC), including allocating compute and storage resources. Tenants/orgs can now provision their own logical network to connect these resources. VCD delegates networking/security control to the vShield Manager, which in turn creates a VDS port group backed by a VXLAN, maps the tenant id to the VXLAN segment id, and connects org VMs to the respective ports in the port group. Additionally, vShield Edge provides multicast services, and maps tenant broadcasts into provider multicasts (using PIM). We now have VXLAN backed logical networks, which are elastic (add/delete vNics/ports on an as-needed basis).

With networking constraints out of the way, VDCs can now span cluster, pod and subnet boundaries, removing one of the major limitations in the data center. The concept of elastic VDCs was an important part of the newly released vCloud Director 1.5.

Two very cool options, which highlight the power of the VXLAN construct and logical networking:

  • VXLANs can span multiple VDS and/or N1K switches (demonstrated at VMworld, using vCloud Director & vShield Manager to federate across VDSes)
  • With VXLAN-VLAN gateways, you can have VMs and physical servers share the same broadcast domain!

Towards secure, elastic hybrid clouds

The elastic VDC constructed above can likewise be instantiated in cloud infrastructure hosted by a VMware Cloud Provider partner. vShield Edge can be used to instantly provision a secure, L3 tunnel to the remote VDC. VMs can now be moved between the local VXLAN and remote VXLAN as needed.

What about securing these VDCs, whether local or  remote? This is where the rest of the vShield portfolio, and our  ecosystem partners come into play. The following is a depiction of the different elements of the solution:


vShield Edge provides perimeter security for the VDC construct. vShield App allows creation of elastic trust zones, whether these be network-centric (break up the logical network into smaller, isolated domains), app-centric (zoning based on Web/App/DB tiers), data centric (zoning based on sensitive data discovery), or identity based (“need to know” access based on Active Directory roles, for example). Finally, vShield Endpoint enables guest VM protection.

It was very gratifying to see Los Alamos National Laboratory (LANL) talk through how they had deployed ALL of the above. Last year LANL was the first government agency to deploy their Infrastructure on Demand (IoD) service leveraging vCloud Director for self-service consumption, and vShield App for micro-segmentation of their VDCs. Leading up to VMworld, they leveraged VXLANs and vShield Edge to extend to a data center hosted at Terremark, building one of the first government hybrid clouds based on the VMware Cloud Infrastructure Suite. Kudos to Anil Karmel (Cloud and Virtualization Architect) and team for standing up these powerful environments.

While vShield Edge, App and Endpoint provide foundational protection and zoning at the perimeter, interior, and VM boundaries respectively, we are working very closely with the networking and security ecosystem (we already have working solutions with Cisco, Trend Micro, and RSA for example, with many more to come) to insert purpose-built functionality at logical boundaries, while seamlessly integrating into the management plane via vShield Manager, which in turn enables these services to be available RESTfully. The combination of logical networking and security, with integrated ecosystem offerings and programmable services, should provide the much needed advancement to support virtualization and cloud needs.


We are seeing a massive transformation in the way networking and security is being re-architected in modern virtualization/cloud data centers. Logical networks, edges and zones abstract the underlying infrastructure, and untether higher layers from the need to be infrastructure aware. The virtualization layer serves to overcome the impedance mismatch between the provider and consumer, potentially unleashing a whole new wave of innovation – secure, elastic VDCs become the new currency for private and hybrid clouds.

Let me end with a quote from Jon Oltsik’s note at Network World:

…VMware’s vision is spot on. Networks are cool and all but ultimately they exist to move data and application bits around. VXLAN is a new way to make sure that applications and networks can do this in a more integrated and efficient way.

Thanks Jon! We couldn’t have said it better…