This is my second CTO blog on VMware and the Internet of Things (‘IoT’). I’ll begin by covering our overall IoT strategy, beginning with the Little IoT Agent (‘Liota’). I’ll briefly review my first blog, Motivating the Three-Tier Architecture.
Many believe that a three-tier architecture will dominate the emerging IoT infrastructure: sensors and actuators (let’s refer to these as ‘devices’ for this post); local systems, that have come to be known as IoT gateways; and data-centers (a catch-all term for public clouds, private clouds, and private data centers). An interesting characteristic of IoT gateways (as opposed to e.g., WiFi routers) is that the communications connection between a sensor/actuator and a data-center de-couples at the IoT gateway (let’s now refer to them as ‘gateways’ for convenience). The connection between a device and the gateway has actual endpoints at the device and gateway (in terms of the seven layer OSI communications abstraction all seven logical connections have endpoints at the device and the gateway) and the connection between the gateway and the data-center also has endpoints for all seven layers at both the gateway and the data-center. This de-coupling allows:
- a physical media bridge
- lower power demands for the attached devices
- lower compute demands for the attached devices
- de-coupling the vastly different expected lifetimes of devices/sw and the expected lifetimes of the web
- a greatly reduced attack surface exposed to bad actors
These facets of gateways, I argue in my first blog, support the idea that much of the emerging IoT infrastructure will be three-tier.
The VMware IoT strategy views this emerging IoT infrastructure as an extension of data-centers, not a special, quirky, new infrastructure requiring significantly different management activities than that of data-centers today. Certainly, the scale in numbers of gateways and data rates might likely be, within an organization, one or two orders-of-magnitude greater, but for the management of the infrastructure some fundamentals apply, e.g., monitor the infrastructure components, raise alerts when issues arise, determine a fix, and apply the fix.
Note that our strategy clearly separates telemetry from devices into two categories: operational and content. Our focus will be on the operational telemetry; doing analysis, alerting, notification, and automatic remediation when appropriate. Much can be said about the details of our approach but we’ll leave it to later blogs. For the remainder of this post I’ll focus on Liota, its purpose, design, and use.
Liota is an SDK and framework, and its purpose is to ease the development of gateway applications for moving data from devices to data-centers and performing commands from data-center components. Liota is a BSD 2-Clause licensed open-source Python project hosted on GitHub and the Python Package Index. Note: Liota is very early in its development and we are bringing more resources online internally at VMware to add functionality. We hope others in the community will join us.
Liota has abstractions for:
- the underlying hardware system (the ‘Board’)
- the hardware plus the operating system (the ‘Gateway’)
- attached devices (‘Things’, although I’m not sure I like this name and we may change it)
- a transformer abstraction, for formatting and/or analysis of metrics
- transport abstractions (e.g., websockets, sockets) for connection to the data-center
- data-center components (‘DCCs’)
- metrics, actions, and alerts
The code for #1 isn’t fleshed-out but, for now, a placeholder. We expect that over time as idiosyncratic functions are found across gateways, appropriate abstractions will be developed. #5, the transport abstraction, is pretty straightforward so we’ll not need to cover it in detail. Suffice it to say we support websockets, secure websockets, and sockets today and will be adding MQTT, CoAP, UDP, and TLS in the near future. The real interesting stuff happens in the remainder.
We’ll start the deep dive with gateways and devices and how they are modeled on the gateway application and in at least one DCC. Think of them as an abstract class with a list of arbitrary key-value pairs, a relationship, a factory for creating metric objects, and a list of actions. Devices have a relationship with their parent gateway, and gateways have relationships with each attached device (child). Relationships are set appropriately by Liota when object instances are created by the application. The application can, after creating a gateway or device object, set an arbitrary number of key-value pairs termed ‘attributes’, in order to create an arbitrary number of metrics and actions. We believe that these four functional pieces of gateways and devices should be modeled in any data-center component. We do so in the one that we have been using to-date, vRealize Operations Manager (vROps), and find it sufficient to perform the management function for gateways and attached devices.
Of course attributes on a gateway object can be used to indicate anything, but the geographical location of the gateway, e.g., Planet:Earth, Hemisphere:Northern, Continent:Europe, Country:France, etc., down to potentially a specific wiring closet in a building, seems to come first to mind. Other uses of interest might be the kinds and version numbers of all hardware and software on the gateway, gateway manufacturer, gateway model and serial numbers, etc. Attributes on devices are even more interesting.
Devices, sensors, and actuators, are correctly called ‘transducers’. A transducer bridges the physical and the electrical. They transform measurable physical quantities to electrical signals, such as a temperature to a voltage drop. Transducers range from very simple, e.g., a thermistor, to complex, e.g., a radiation detector. All transducers have what is called the ‘data sheet’. For simple transducers the data sheet may be a few pages, but for complex transducers the data sheets can become very long. The data sheet explicitly states every possible characteristic, range, connectivity, etc., necessary for an engineer to fit the transducer into the design of a larger system. We believe that for industrial, accurate, and precise analyses many of the values in the data sheet must be available to the analytics logic. Thus, we see attributes of devices managed by a Liota application as the appropriate place to encode necessary values from the transducer’s data sheet. The IEEE has a specification (1451, Electronic Transducer Data Sheets) which is on our list to support in future versions of Liota. Device attributes may also hold location, manufacturer, etc., as gateways.
Liota abstracts the data-center components in the DCC classes. These classes hold the protocol and formatting logic for a particular data-center component and can be very simple (Graphite) or complex (vROps). Simple DCCs typically accept a stream of values with associated timestamps and names. Graphs and analyses are built from the incoming data. Such data may also be moved to persistent storage for offline/batch analysis or visualization. We believe that these simplistic data flows will eventually prove insufficient to realize the greater promise of IoT, and that meta-data, integrated command flows, and alerts will enable more sophisticated IoT solutions.
We have been using vROps, a data-center monitoring tool, to experiment with a more complex IoT infrastructure management solution. I’ll mention here that our model is somewhat different than one you would typically find in emerging IoT management (and even data-center management) solutions. Our model puts the burden of the creation of the representations of gateways, devices, attributes, alerts, actions, and metrics both on the IOT gateway and in the data-center components on the gateway application rather than on the UI of any particular data-center tool. In this model vROps is a simple framework initially, knowing nothing about any gateway or its attached devices. It’s the responsibility of the gateway application to create a complete representation of itself and its attached devices, including all necessary meta-data in the data-center component. The underlying reasoning for our model is that the gateway application is the place-or-origin for everything related to the gateway and its attached devices, and thus the gateway application is the piece of logic that is best positioned to create and maintain these representations and functions.
Now let’s look at metrics in Liota. A metric object represents a stream of numerical values related to either a device or a gateway, obtained from anywhere, and flowing to a particular DCC. An application creates a metric with a method on a DCC object. Parameters include: the device or gateway object, a name, an SI unit[1], a sampling interval, an aggregation count, an arbitrary string representing the device type, and the name of a method (we call this a user–defined method or ‘UDM’). Upon initialization, metric objects are placed into the Liota framework and operate according to the values of parameters given in the creation method. The UDM, given as a value to the sampling_function parameter, is called at intervals given by the sampling_interval_sec parameter. The UDM should include a communications endpoint from which it can read a value from a sensor and return a scalar numeric value. The framework adds a timestamp and collects a number of values given by the aggregation_size parameter. Metrics can either poll a data source or wait on a data source for data to arrive (event model). For a polling mode the UDM should assume that a value is ready and return immediately. To establish an event metric, set sampling_interval_sec to zero and code the UDM to block until the datum becomes available. UDMs can contain arbitrary code and are limited only by the developer’s imagination.
For example, consider a metric with a sampling interval of 100 seconds. The UDM could certainly include a one-second wait and collect, filter, smooth, analyze, etc., 99 measurements and return the computed result. At the 100th second it will be called again by the framework and the logic repeated. Using the event model with the sampling interval set to zero, the UDM could spend an arbitrary amount of time collecting and analyzing values from the sensor and return at an appropriate time. The framework will then call the UDM immediately so it can repeat. Metrics represent the flow of data from a device to the data-center. The inverse, commands from the data-center to a gateway or device, are realized by actions.
At this point the version of Liota on GitHub does not include fully implemented actions or alerts but I’ll describe how they will work. They are our next priorities for Liota. Creating an action mirrors the creation of a metric. Parameters include a name, a key, and a UDM. The name and key are sent to the associated DCC. The DCC displays the name for either user selection or auto-remediation. Upon selection the DCC returns the key to Liota and Liota uses that key to call the associated UDM. The UDM is free to do whatever logic is appropriate to complete the action. This logic can range from updating firmware on a device, to upgrading an application, to rebooting the system, to performing OS clean-up function, or more.
The application will also be able to create alerts in the DCC. We haven’t fully fleshed out alert creation yet, but we are planning to create them in a similar fashion as the metrics and actions. An alert will be associated with either the gateway or an attached device, a DCC, and be based on one or more metrics or attributes. Creation will include parameters for a name, thresholds of metrics, values of attributes, and values or thresholds for functions of different metrics, among others. Alerts at the DCC serves two functions: notifying business applications and/or personnel not available near the gateway, and alerts that may depend upon data not typically available at a single gateway, e.g., temperature threshold and the weather forecast for the location of the gateway and devices.
Developers use Liota by creating an application using the abstractions, objects, and methods in the framework. A typical program creates local representations (objects) for the gateway, any attached devices, a transport, and a data-center component. For some DCCs these representations can be also created within the DCC. The program will then create metrics, alerts on those metrics, and actions. Now the program causes the metrics to start collecting. Any actions from the DCC will be handled by the appropriate UDM.
Please consider joining the Liota community and help us create a great gateway SDK for IoT! Next time I’ll discuss the larger VMware IoT strategy, including some upcoming new tools.
[1] See, https://en.wikipedia.org/wiki/International_System_of_Units , an SI unit specifies the type of physical quantity being transferred, e.g., m/s2 for an acceleration.
