RuBAN server side data management

This page details the architecture for the RuBAN data flow architecture specifically in the cloud. It details the data input sources into RuBAN, flows through RuBAN and the flows out of RuBAN.

Data sources

There are three input data sources to the RuBAN platform. IoT Gateway, Web enabled API’s, legacy data sources. An IoT gateway device is under the category of an intelligent gateway. Examples are the Cisco 819, IR 910, 829. RuBAN embeds agents inside the intelligent gateway which harvests data from sensors and pushes to RuBAN. The second data source is Web enabled API’s. Many systems open up their data through open API’s, RESTful or SOAP interfaces are common. RuBAN can quickly add connectors to these systems to get their data into the RuBAN database. Legacy data sources are typically expose their data through an SQL, OLAP or even a file system (e.g CSV, XML). The RuBAN platform can take this legacy data into the platform, usually to eventually combine with some new data source to enable better data for the user. Typically data is imported in bulk from these systesm using an ETL (Extract, Translate, Load) intermediary system. RuBAN is not tied to using any particular ETL system but has been deployed most commonly with Pentaho Kettle.

Input data bus

The input data bus needs to be flexible to accommodate the different data input methods. Data can be sent into RuBAN using different transport methods. In different deployments and with data with different levels of importance, different transport mechanisms will make more sense. E.g UDP if data can be lost, TCP when there needs to be reliability, SCP for a secure bulk upload etc. RuBAN has been architected to allow for these data input methods to be enabled side by side and any new input methods required by the customer can be added as needed if not already covered off.

Storage

Headers and footers are stripped off the data and the payload is passed onto RuBAN’s TSDB (Time series database). The TSDB is a wrap around on Cassandra. Cassandra is a NoSQL big data store. The TSDB marshal’s data going into Cassandra and data being queried out of Cassandra and ensures that it is written in a format such that it can be written in real-time and read in real-time.

Data that is non time series data can also be stored on the RuBAN server. This is often data that is supplementary to the time series data. For example descriptions of dimensions, sensor semantics etc. Configuration data etc.

Open API

RuBAN comes with a standard set of API call which are fully documented on Davra’s resources section of the website: http://dev.davranetworks.com/api/v1/ Here you can get IoT gateways or entities that are existing in the system, query what sensors are associated with those entities and then query to get the raw sensor values. There are many other API calls to expose all the other areas of the platform from provisioning devices in the field right to delivering data to the user.

The standard API can be extended. It is not uncommon for raw sensor values to not give useful info on their own. Using RuBAN SIDE (System Integrator Development environment) the user can create aggregations or even join sensors to create new more useful API calls. An example is useful to illustrate this. For a vehicle it is common to monitor idling. This is when the vehicle is stopped but the engine is ticking over. There is no sensor on the engine that gives idling. You do however get speed and rpms and from these you can infer idling. Using SIDE you can combine these 2 sensor values and create a new API call that gives you idling over time. For each point in time if speed was not detected and RPM’s was less than 800 then idling was 1 otherwise it was 0. This can then be pushed back into RuBAN’s TSDB and is automatically wired up to RuBAN’s dashboard as a new API call which can be charted and visualised. Custom aggregations can also be performed over the data and stored in RuBAN’s database.

Application builder framework

Users can use the application builder framework in RuBAN to build applications on top of the RuBAN platform. Using a drag and drop framework to create layouts, drop relevant components into the layouts, wire up the components to the sensor values. Applications can also be styled and published and made available to used again.

The application builder can consume data from the OPEN API through the RESTful API, web sockets or streaming data. Restful API is the most typical from the data is consumed. This makes a HTTP request to RuBAN’s API and a JSON response is returned. The component then consumes the response and display the data. Websockets can be used for a constant connection perhaps where a more real-time component needs to be displayed. Streaming data is typically video. RuBAN re streams video from the VSOM through to the RuBAN dashboard where it can be viewed.

Data output bus

All of the data transport methods available to get data into RuBAN are also available for 3rd party systems to get data out of RuBAN. So this is entirely flexible based on the system you need to send data to. The system has been architected so that new transport methods can be added as needed if they are not already readily available.

Depending on the system you are sending data to there may be different formats that it needs to be sent in. For example in one scenario tested data was taken out of RuBAN in avro format where it was pushed to the HDFS in Cloudera and then an Impala layer was put on top so that it could be easily extracted through a BI system like Jaspersoft. In another example the GPS data for a fleet of vehicles was made available through Google Protocol buffers adhering to GTFS so that it could easily be consumed by iPhone applications. New serialization formats are being added all the time and the system is architected so that this will be a simple operation for the user.

New Data aggregations

Data aggregations can be set up over the raw data. Sometimes for certain reports it will make sense to run the report over aggregated data. For this we need to allow the user to create their own custom aggregations on the fly. These aggregations are automatically wired into the RuBAN API and usable through the RuBAN UI.

CEP

CEP operations can be defined to run over the flows to do complex event processing. This is where data can be inspected in real time in whatever fashion is suitable and triggers alerts to other systems. These operations can be added on the fly to the RuBAN platform. An example is if event a is triggered and event b is triggered less than 20 seconds later then escalate this via event c. RuBAN has in built CEP but can also integrate with enterprise CEP platforms such as Oracle CEP and Esper CEP.

Third party API push

RuBAN can push information to other systems in whatever format they expect. This may not be a big data system it could be an SMS, an email, or send a SOAP message to create a job in some job ticketing system. The RuBAN platform is adaptable to fit into the workflow of the customer. Whether that is telling a mechanic to order a part for a bus, send a message to a drivers in vehicle display to let him know a bin is ready for collection or to warn a police officer that a suspected assault is occurring on a bus. These can be added flexibly to the system.

Big data systems

RuBAN will continue to add support for third party big data systems. Recently connecting to Cloudera. RuBAN has been validated and certified as compliant with Cloudera by the Cloudera engineers: http://www.cloudera.com/content/cloudera/en/partners/partners-listing.html?q=davra RuBAN also works with DataStax and will continue to add more systems over time.

Browser not supported