One of the most overused terms of the past couple of years or so have been Big Data and the Internet of Things. The amazing new database technology that companies like Google, Yahoo and Facebook have developed to handle large data sets have contributed to take the concept of Internet of Things out of the realm of science fiction and into the sphere of possibility. For the ones of you who have heard the term but never had the chance to understand what it means; Internet of Things refer to the interconnection of computing like devices with the existing internet structure.
The ultimate vision behind the concept is for smart sensors and other embedded devices to be hooked up to the world wide web, allowing for an unprecedented level of automation. From washing machines that would by itself schedule a service call when it detects it is malfunctioning, through bio-chip transponders on farm animals constantly taking measurements and storing them on a central database for animal health monitoring; and heart implants that would call 911 in case it detects an emergency in the heart it is implanted to.
It is easy to see how an amazing amount of data is generated by this mass of intelligent devices hooked up to the web. The conventional network hardware couldn’t possibly keep up with this demand by utilizing the same databases and data warehouses of the past. Very quickly, the amount of data generated by just a subset of these smart sensors would make the conventional solution not to be viable. However, the so called Big Data paradigm was developed, which brought the Internet of Things dream several steps closer to reality.
Since we are involved in the Test and Measurements industry, it is natural for us to ask the question: How will Big Data and Test and Measurements play together, if at all? It is very easy to see how enterprise management systems are getting more and more integrated down to the manufacturing floor where test systems are deployed, collecting data. Also, quality systems are being integrated with characterization systems for product development activities. It has become all about the data these days.
Added to that, the improvements of data acquisition hardware has delivered to the public acquisition devices collecting data on the mega sample per second and starting to migrate into the giga samples per second realm. This has contributed for scientists and engineers to become even more data hungry than before.
All of this has launched National Instruments onto its own pursuit to integrate Test and Measurements with Big Data. NI has coined the term Big Analog Data. Big Analog Data is basically Big Data derived from the physical analog world, i.e. data collected by acquisition devices.
This is certainly great to see as National Instruments usually contributes with cutting edge innovation that helps the solutions to these types of problems to come to light. However, in my humble opinion, I do think NI may be approaching the problem from the wrong angle. NI’s Big Analog Data solution is focused on the large mainframes and conventional IBM-type of hardware infrastructure. As I mentioned at the beginning of this blog, Google, Facebook and Yahoo solved this problem by creating database technology that made cluster of distributed inexpensive computers as powerful or more than the good old several hundred thousand dollar mainframes of the past. It made large data sets to be financially viable by solving the problem in the software domain, not in the hardware domain.
In my opinion, the answer for Big Analog Data should be aligned with the success stories of the three giant web companies I mentioned here. The path should be the creation of database solutions that would align well with the giga sample per second multi channel devices that would work on inexpensive cluster of servers.
Google, Yahoo and Facebook already solved the problem to best fit their industry; which I will call here Big Slow Data. Internet type of data that could be refreshed once a second or slower. The leap that needs to be made for Big Data to truly integrate with Test and Measurements is the expansion of this paradigm onto a database solution that would support data to be stored in the giga sample per second rate, while also allowing for queries in parallel to data storage. This would put the power of Big Data in the hands of start-ups and other small companies, as well as would make it economically viable, throwing gasoline in the Internet of Things fire.