Lead Data Operations Engineer
MarketAxess
The Internet of Things (IoT) is a network of connected devices that can communicate with each other and exchange data. According to a forecast by Statistica, the total data volume generated by IOT devices worldwide is expected to reach 79.4 zettabytes (ZBs) by 2025 (Source: www.statista.com). There is tremendous potential in what this data can do for our businesses and organizations, however a large portion of this data goes completely untapped. Today, we will explore some of the challenges in IOT data storage and the techniques used to analyze it efficiently.
IoT Data Storage Challenges:
The main challenge with IOT data is that it is largely unstructured and comes from a variety of distinct sources. For instance, an agricultural IOT device sends temperature metrics, fertilizer stats, moisture levels, plant images and videos pertaining to a plant every minute. The format, metadata and metrics associated with the above types of data, are quite distinct from each other and cannot be molded into a standard data structure.
Conventional data storage systems like SQL Databases are rigid and may not be well suited to handle this type of data. We need a storage system that is highly scalable to store the ever-increasing volume of data, can store multiple data formats (audio, video, blob, etc.) and provides us with fast querying and analytics capabilities.
Choosing the right Data Storage solution:
There are various solutions available to store unstructured data like cloud data lakes created on Azure Blob Storage or Amazon’s S3. You can use No-SQL Databases like MongoDB, Cassandra DB, time series databases like Graphite, InfluxDB or utilize cloud data warehouses like RedShift, Snowflake, etc.
But when it comes to IOT data, we need to find a solution that can efficiently store heterogenous unstructured data, as well as keeps incremental costs of high volume IOT data in check. The two main use cases that businesses and organizations require IOT data are:
- For data analysis and predictive modelling
- Monitoring and observability
For the first use case (data analysis) we need to store data in a long term, cheap and durable storage repository. Whereas for observability, data is short lived and is often needed in real time. Thus, we need extremely low latency short term data stores for storing and retrieving it.
One popular strategy to tackle this, is to split the IOT data into two zones – a long-term historical zone and a real time (short term) data zone. For long term storage, we can use cloud-based data lakes that offer a very cost-effective and flexible approach to manage, store and retrieve unstructured data. Most cloud platforms provide data lake storages like Amazon S3, Microsoft Azure Blob Storage or Google Cloud Storage ensuring high durability and availability. These are equipped to store enormous amounts of raw data in its native or processed forms, allowing organizations to run big data analytics as well as providing the option for data transformation and integration with various tools and platforms.
For the monitoring and observability needs, we can use a durable pub-sub streaming platform like Apache Kafka. Using real time data stores like Kafka, IOT data from millions of sensors can be stored and propagated in extremely high speed (sub milliseconds) to a multitude of platforms. These platforms can then utilize these real time data for monitoring, alerting, or taking corrective actions, etc. Many cloud providers like Amazon AWS, Microsoft Azure, Confluent Cloud etc. have their own versions of Kafka that can be seamlessly used in integration with a large number of applications.
Data Transformation and Analytics for IoT
Coming up with an optimal data storage is just half the battle. In order to efficiently store and retrieve large amounts of unstructured IOT data, we also need smart data processing, cleansing and transformation pipelines.
This is where distributed computing systems for big data processing come to the rescue. Distributed Computing systems use technologies like map-reduce and splitters to split processing jobs into very small batch of jobs that are distributed across a cluster of compute nodes to help transform large amounts of raw data into refined data at extremely high speeds. Apache Hadoop and Apache Spark are two of the most popular Big Data processing frameworks widely used across the industry for this purpose.
Refer the diagram that showcases a simplified form of an IOT data pipelines (1a. Example of a simplified data pipeline)
In the above diagram, we have used Amazon AWS cloud services, but most cloud providers like Google Cloud, Microsoft Azure, etc. provide very similar data storage and analytics services. The first stage of the pipeline is collection stage where we ingest raw IOT data from various sensors, cameras and other IOT devices into an Amazon S3 object store, called as our Landing zone bucket. Next, a distributed computing service like Elastic Map Reduce(EMR) is used which processes, cleans, reduces and extracts the most useful portions of this data for analytical consumption. An EMR service can be used with various big data processing frameworks like Hadoop, Spark, Hive, Pig, etc. to facilitate this.
This transformed data is then partitioned and stored in a refined zone S3 object store, typically in a format like csv, json, parquet, avro, etc. We can then use various analytical engines like AWS Athena, Tableau, etc. to directly query this IOT data in SQL query language to retrieve data, create dashboards or process it as per our business needs.
Smarter IoT Analytics using AI
Today, there are multiple AI frameworks like rule-based AI, neural networks, machine learning and generative AI that can be utilized for different business use cases. For IOT data, ML-based technologies, including natural language processing (NLP), image recognition, audio analytics are critical to uncovering hidden data and insights. Natural Language Processing (NLP), a branch of artificial intelligence, is primarily used to analyze text-based unstructured data. NLP techniques like Text Classification can be used to efficiently categorize data in cloud data lakes. NLP also offers Intelligent Document Processing and Information Extraction techniques that use computer vision and intelligent extraction to classify and organize data in our data lakes.
Other AI techniques like Image Analytics, Video analytics can identify critical information from videos, images coming from the IOT cams for automated data extraction and generation. Creating Machine Learning models is a popular technique used across industries to process and analyze different types of unstructured data, such as text, audio and images, which can be used to build and implement AI models. There are ML tools like TensorFlow employing deep learning algorithm, and IBM Watson providing NLP, sentiment analysis using pre-built ML models that can be integrated in your analytics platforms.
In conclusion
This article has provided a glimpse into the vast and exciting world of IOT data storage and analytics. There are numerous such frameworks, tools and technologies that can be leveraged to transform complex IOT data into meaningful data patterns and AI predictive models. These models can provide organizations with critical business insights to stay ahead of the competition and make informed decisions. As the fields of IoT and AI mature, it is imperative that we start harnessing their potential today, because the convergence of these technologies can revolutionize the way businesses operate and innovate in the future.
