Chris Wolf and Andrei Warkentin
Today’s VMworld US keynote included a demo where we ran four hypervisors concurrently on the same x86 server, and none of the hypervisors were nested. While on the surface that might seem impossible, the magic that made it all happen was our continued work on running ESXi on 64-bit Arm platforms. The demo featured ESXi running on the main x86 platform and ESXi-Arm running on three plugged-in SmartNIC adapters (Mellanox BlueField and Broadcom Stingray). One SmartNIC ran the Snort NIDS, logging suspicious traffic to a local database for further analysis, while the other pair ran an SD-WAN virtual network function (VNF) VM in fault tolerant (FT) mode.
This is the high-level architecture.
In general, our work aimed at running ESXi on 64-bit Arm platforms is focused on a number of use cases, including:
- Cloud-hosted virtualized Arm – Arm-based cloud for cloud-native workloads and other customer use-cases such as Android test/dev environments
- Edge consolidation and isolation (e.g., SD-WAN, IoT gateway, plus apps)
- Edge resiliency – High Availability and/or Fault Tolerance for Arm-based applications at the Edge
Specific to SmartNICs, we are starting to see some early use cases emerge:
- Management and/or data plane offload
- Tenant isolation to a single SmartNIC (no PCIe hop)
- Energy efficiency
As for the demo itself, we ran our hypervisor on the following SmartNICs:
- Mellanox BlueField MBF1M332A-AENAT
- 8 Cortex-A72
- 16GB RAM
- 2 x 25GbE SFP28 via ConnectX-5
- Broadcom Stingray PS225
- 8 Cortex-A72
- 8GB RAM
- 2 x 25GbE SFP28 via NetXtreme
The host server ran ESXi with a vCenter VM and served two iSCSI targets and two NFS data stores. The NFS data stores were used for HA between the Mellanox SmartNICs and contained the VMs running on the SmartNICs.
The Broadcom Stingray ran a “stateless” ESXi-Arm image, contained on the on-board eMMC (no persistent storage and configuration changes across reboots), with the NIDS VM stored on an NFS volume.
The Mellanox systems were full ESXi-Arm installations over iSCSI targets exported by the x86 host, with the VNF VM stored on an NFS volume.
Booting differences aside, the ESXi image running on either SmartNIC was exactly the kind of generic ESXi-Arm image that we have demonstrated in the VMworld 2018 Arm Virtualization at the Edge keynote demo, or that you could run on an Arm server system compliant with the industry hardware (SBSA) and firmware (SBBR) standards.
For the demo setup, all VMs were Ubuntu 18.04 LTS VMs. We can run other Linux distros such as CentOS, SuSE, Fedora and FreeBSD.
VMware’s Office of the CTO, in partnership with our Cloud Platform Business Unit, continues to explore emergent Arm use cases with the goal of bringing a solution to market. We continue to demonstrate the art of the possible as we pursue the right use case to bring a solution to market. Have a compelling use case or opportunity? Let’s talk.
