Worldwide data volume is growing exponentially. At the same time, the need for data-assisted capabilities supporting the enterprise is becoming vital. Combined, these factors present an interesting and complex challenge.
The technological part of the solution is anything from trivial. There is also a variety of possibilities for the operational model of the system. Last but not least, before we jump into implementation, we must define system capabilities.
Out-of-the-box solutions offered by hyperscalers may sound tempting. But hyperscalers are expensive — especially when your system matures and it is already too expensive to migrate away. Also, while it might sound like a safe bet to lean towards the traditional centralized operational model with a large team of data engineers, this model lacks the agility needed by the dynamic world of business.
Like most large enterprises, VMware needed to get more value out of its data by removing data silos and combining disparate data sources into a centralized and governed big-data system. After spending years designing, building, operating, and evangelizing such an analytical system, we came to believe that operating a private cloud based on a VMware stack and using OSS is the technological approach that gives agility at a good price.
For this approach to be sustainable, the interfacing surfaces provided to system users should be very carefully selected, so that it is possible for the underlying technical solution to be changed transparently to (or at a very minimal cost for) the users. To maximize the potential of the data, the “self-serve” model is a key step to achieving data democratization.
After having invested over 100 person-years in this endeavor, our experience has shown us that users of the system are most effective when organized in data-mesh-like teams. Additionally, their efficiency is highest when the system is self-service. The data platform we deployed — known as Super Collider — achieved an industry benchmark: a net promoter score of 81.8, qualifying it as a “best-in-class” offering.
Read more details about what we’ve built and why we’ve built it this way in our new whitepaper “A Best-in-Class Data Analytics Platform, Built on a VMware Private Cloud.”