Virtualizing Big Data

Analysis of large-scale, often unstructured data is becoming increasingly important within both the Enterprise and the HPC community. This is perhaps one of the most apparent areas where the convergence of HPC and Enterprise requirements can be seen as the tools and algorithmic approaches required are often the same or very similar. I imagine, for example, that the large-scale, graph-oriented “social network” analyses done by companies like Facebook are quite similar to the “anti-social network” analyses done by Homeland Security and the Intelligence community.

Unsurprisingly, many VMware customers are interested in running Big Data workloads and are looking for guidance about how best to do this in a virtual environment. To help, we have published a whitepaper that examines Hadoop performance using local storage in a vSphere environment, the first in what will eventually be a series of whitepapers in this area. The current paper is available here.

For those interested in a broader discussion of Big Data, NOSQL databases, and virtualization I recommend an audio recording of the Big Data panel that was held at VMworld in Las Vegas. Our panelists were luminaries from across the Big Data space: Amr Awadallah, CTO Cloudera; Clint Green, Principal Engineer Data Tactics; Paul Kent, VP Platform R&D SAS; Luke Lonergan, CTO Greenplum/EMC; and Richard McDougall, Technical Architect for Big Data, VMware. It was a real treat to have all of these experts together in one panel session.

The audio is available here (free registration required).

Other posts by

vSphere Scale-Out for HPC and Big Data

I’m very excited that we’ve announced vSphere Scale-Out this week at VMworld here in Las Vegas. This new vSphere edition is specifically and exclusively designed for running HPC and Big Data workloads. This is an important development in our work to offer compelling virtualization solutions for these two emerging workload classes. Our strategy for addressing […]

Three Extreme Performance Talks from the Office of the CTO at VMworld USA

The Office of the CTO will be presenting three talks in the unofficial “Extreme Performance” series at the upcoming VMworld 2017 conference in Las Vegas. In addition, one of these talks will be delivered at VMworld Europe in Barcelona. Each of these talks focuses on important aspects of pushing the envelope to achieve high performance […]

How to Enable Compute Accelerators on vSphere 6.5 for Machine Learning and Other HPC Workloads

As our CTO Ray O’Farrell recently mentioned, VMware is committed to helping customers build intelligent infrastructure, which includes the ability to take advantage of Machine Learning within their private and hybrid cloud environments. As part of delivering this vision, the Office of the CTO collaborates with customers and with VMware R&D teams to ensure the […]