Generic IoT Platform Thoughts
With few years of experience in IoT, we had played quite well in IoT and Azure landscape.
- Worked With 400K Devices around the globe, which are sending aggregated information on every one hour
- Worked with UPC-UA ingesting around 300Million Data points per day
- Near real-time processing
- Historic Reporting
- Business critical application to experimental applications
Now we are planning to review the way we progressed and see if we can create a Generic IoT platform.
Kind note that the entire view is specific to our use case and landscape and the same may be different to others
Please find the IoT progression graph ( based on Microsoft documentation) below. Now we have almost finished the phase 1 and looking forward to next levels. That’s where we are planning to review our progression till date and validating a generic IoT platform will suit us.
Is a generic IoT Platform good for us?
Yes, only if you are having control over the devices/edge then its a must have. Else, it will be possible, But its having some limitations. Lets discuss those views detailed below.
You can find some interesting links in reference section below, which explains how difficult it is to have a generic IoT platform.
In a typical IoT environment the data flow will look like below
Sensors will send data via edge devices to cloud and later the data will be passed on to applications.
And if you see the basic processing happening on both cloud and edge will look like below
Until this point it looks fine and we should create a platform to serve for the IoT needs of the organization and process similar data. Wait a minute and lets look deep into the same. Based on our analysis each step can be grown to a huge list if the organization has a wide range of devices in place. lets look into each of them below, will do some exercise after going through them
Data
There are multiple type of messages can be sent from devices, as all the devices work with different protocols and functionalities. Please find some samples below, some of them may be overlapping.
- Time Series Data — Data points coming as time series, This method will be chosen if the data points are independent and the data flow should be as real-time as possible
- Aggregated Data — Data points coming as an aggregated value for a period of time, This method will be chosen if we are more interested in aggregated data when the criticality is less and transmission constraints are there.
- Log File — Data points coming as large log files containing all telemetries in one file
- JSON/Protobuf/OPC-UA
- and many more
Now we can simply see how the data become diversified when we look more closer, this will mostly happen if we considering an organization who uses other vendor devices for their IoT needs.
Communication
Edge will be communicating to its higher level using many protocols and methods and lets see some of them below. Am not going deep into them now.
- MQTT
- Kafka
- OPC-UA
- Lora
- Lora Wan
- Coap
- Http/Https
- Proprietary Protocols
- Wirepas
- and more
Processing
When comes to processing, it looks straight forward, but lets look into the some of the common processing steps below
Telemetry Processing
- Device to Cloud Messages
- Cloud to Device Messages
- Cloud to device Response Messages
- Alarm Messages
- Log Messages
Device Management
- Device Provisioning
- Firmware/Software upgrade
- Security
- Device Decommissioning
Integrity Checking
Encoding/Decoding
Compression/Un-compression
Aggregation
and more based your use case
Persistence
Based on your need, volume and type of data, persistence of the same will also be varied. Please find some common persistence below
- SQL
- PostgresSQL
- TimeScaleDB
- Azure Data Explorer(ADX)
- Cosmos
- Casandra
- Mongos
- and more
Application
Now once we persist and share the data to end users, the visualization and application landscape will also be varied. Please find some of them below
- Enterprise application built with Asp.Net, Angular, ReatJs,
- Reporting applications like Power BI, Grafana
- and more
Now lets write all your component and try to find the overlapping items among the same.
Once you found that your use cases overlaps on these components, you can proceed to build an IoT platform else the same will be less effective than you imagine.
Now if you have found that there is some overlapping capabilities and you are proceeding to next step. Lets identify the platform capabilities which are more related to IoT and that can be shared between different use cases or products within your organization.
Platform Capabilities
Lets find some sample capabilities which we can consider for the platform
Now we are adding some more capabilities which can also be considered to enrich the platform which is more of your organizational capabilities.
Now we have added few more to the same (highlighted in gray).
Now lets wait a minute and take a pause. Is this categorization of cloud and edge is still valid ? Lets answer that and come back to this again.
In this year 2022, edge is getting more and more capable and the application footprint on the edge is also getting larger, so you have to classify which category your assets are belonging to
One you have done the classification and if your edge devices are capable to process more, then you will probably answer the question “Why we are not processing on edge?”
Now if you look back the previous capabilities which we discussed, you can see most of them can run on edge if its capable (highlighted with green).
So in that case we can create capabilities which we should run in both edge and cloud and lets proceed to build the same
You can refert to other article related to same for more link
Conclusion
Now as we can see we can identify some IoT relevant and specific capabilities which we can consider as part of the platform and all the product level capabilities should be part of the products.
Now we have brainstormed well for your platform and lets build the capabilities. While building the capabilities, we should build capabilities which can run both on cloud and edge seamlessly.
You can also look in to similar products like openremote for reference
For that you can also read my upcoming article, which will help to create a cloud agnostic application, so that it will run on both.
Please share your thoughts on the same.
Reference links
