Hadoop Performance on vSphere 5.1

Hadoop logoWe’ve just published a third Hadoop performance paper, written by VMware performance expert Jeff Buell, which looks in detail at the relative performance of a bare-metal 32-node Hadoop cluster compared to a range of virtual clusters with up to 128 VMs. The executive summary is that while we saw a 13% performance degradation in a head-to-head comparison of a 32-node physical cluster against a 32-VM virtual cluster (one VM per host) running on the same hardware and running the same tests, virtualized performance can be increased significantly — to the point where virtualized Hadoop actually runs a bit faster than physical — by increasing the number of VMs per host. We’ve seen this effect before with Hadoop and with other resource-intensive HPC applications.

Read the full paper for detailed results and to learn about performance best practices for deploying Hadoop on vSphere.



Other posts by

High Performance Computing with Altitude: SC’16 Begins Tomorrow!

As readers may know, VMware has had a presence in the EMC booth for the last several years at Supercomputing, the HPC community’s largest annual ACM/IEEE¬†conference and exhibition. With the fusion of Dell and EMC into DellEMC and with VMware now under the Dell Technologies umbrella, I am very pleased that we will have two […]

Performance of RDMA and HPC Applications in VMs using FDR InfiniBand on VMware vSphere

Customers often ask whether InfiniBand (IB) can be used with vSphere. The short answer is, yes. VM Direct Path I/O (passthrough) allows InfiniBand cards to be made directly visible within a virtual machine so the guest operating system can directly access the device. With this approach, no ESX-specific driver is required — just the hardware […]

Virtualized HPC at Johns Hopkins Applied Physics Laboratory

Johns Hopkins University Applied Physics Laboratory¬†(JHUAPL) has deployed a virtualized HPC solution on vSphere to run Monte Carlo simulations for the US Air Force. The idea was conceived and implemented by Edmond DeMattia at JHUAPL, and has been presented by Edmond and his colleague Michael Chinn at two VMworld conferences. We now have a white […]