This article is for all the distributed systems application programmers out there who want to increase their programming efficiency; not have to worry about the complexity of the tool they are using; increase throughput and reduce the IT complexity for the end customer. You know who you are, the person who is now going through a database tool re-evaluation to find one more suited to your needs.
The VMware NSX-T team found themselves in this situation almost two years ago when they decided to replace their underlying database for the management plane that is responsible for managing a logical network topology for a Software Defined Network (SDN). The management plane cannot tolerate any failures and must be available to serve transactional APIs to update the several entities of the logical network atomically. To come to a point, it is important.
An evaluation team, which included members of the VMware Research Group (VRG), looked at the options and concluded that all fell a little short. Although the options stored the data securely, the application itself had to keep storing and retrieving the data in/out of the databases in a format they dictated and was hard to manipulate. They also required the customer’s IT team to install and maintain the database in their IT infrastructure.
The VRG team threw into the mix the possibility of using Corfu, a VRG research project that is built on the simple idea of using in-memory distributed datastructures backed by a shared log. Say What? Yes, they proposed using a database-less database. The evaluation team built an initial prototype and ran workload tests against it to validate this approach. The results were promising and showed Corfu was a viable option. But as you know, getting buy-in to use a new data technology in an existing product usually produces more hurdles than the technology itself. Keeping with VMware’s innovation-friendly culture, the VMware NSX management team embraced the disruption.
Okay, you are wondering, “How did it go?” and “How hard was it to implement?” Simply stated, it went very well. The NSX-T developers were able to stop worrying about the distributed system underneath and store their data needed for the management plane in Java objects. Something as familiar to them as riding a bike. The developer was in charge of the data model since the data is stored on the application side. Corfu is managing the state of the system by storing transactional, persistent, replicated objects over a distributed shared log. A history of updates represents the state of each object. The log is doing all the hard work of ensuring reliability, durability, and elasticity. The beauty is this works for multi-threading too. The ability to update multiple objects atomically was a perfect fit for the SDN Management Plane. Oh, and by the way, this took just about a year from prototype to completion with a small compact team.
So Corfu is asking you, will you take a chance on me? I am highly scalable, fault tolerant, self-healing, strongly consistent, and can sustain 0.5 Million ops/sec. Did I mention, I am deployed as a single service with no on-going administration, so IT costs are lower than a non-database-less database. If you are interested contact vmwresearch@vmware.com to learn more.
