The Internet of Things (IoT) brings the promise of new possibilities, but to unlock them, organizations must change how they think about data. With the emergence of the IoT, we are seeing a new class of applications that deal with streaming data coming in from a wide array of sensors and devices across the globe. In essence, the world of IoT involves a paradigm shift from today’s static and finite data to the world of real-time applications consuming continuous and infinite data streams. With this new model comes new demands that quickly overwhelm today’s legacy infrastructure stacks.
New Demands in the Streaming World of IoT
- Taming the data stream: IoT data coming from a sea of global devices takes the form of streams, which are inherently different from the static data that we are used to. Streaming data is continuous and infinite, and it comes fast. It has a state, but arrives out of order, transmits multiple times, and may have large chunks missing. To take advantage of the insights contained in streaming IoT data, next-gen solutions will need to be able to navigate these distinctive characteristics to make streaming data analytics reliable, durable and consistent.
- Volatility to the max: The volatility challenge of big data is exponentially greater with streaming. Gone are the days of consistent, nightly batch loads. IoT data is always in motion, and its volume is in flux at any given time. Therefore, the need to auto-sense and elastically scale up and down to react to dramatic peaks and valleys within the streaming workload is critical.
- Infinite scale – The sheer volume of devices and exponential speed at which data is created brings an exabyte-level data scaling challenge. Before IoT, most of this data was just thrown away, but now you need a plan to process and a cost-effective way to persist, enrich and feed all of this data to your data science teams.
- Challenge of securing accurate real-time insights: In the world of IoT, time is the difference between acting on an insight to add business value or missing a critical opportunity. Moving to true streaming analytics engines can cut valuable minutes and seconds off of traditional mini-batch analytics approaches. Taken a step further, moving these analytics out of the cloud and onto powerful servers in the fog (the edge in some cases) can provide even more timely and accurate insights.
Today’s Do-It-Yourself Solutions
IoT represents a huge opportunity for businesses, but putting it all together has been a challenge for organizations due to the demands detailed above, which traditional solutions were not designed for. In order to achieve a semblance of ‘stream-like’ capabilities, today’s solutions resemble a patchwork quilt with disparate technologies stitched across a myriad of software venders and hardware stacks. Early adopters have struggled with these DIY solutions which are costly, complex, and offer limited applicability. The solutions have redundant siloes for each separate function and then complicate it further with individual data pipelines for real-time, batch, and disaster recovery. And at the end of the day, they still don’t fully achieve their goal: actionable real-time insights.
Delivering the Platform Necessary for IoT: Project Nautilus
At Dell, we see an opportunity to radically simplify the DIY infrastructure stack that has come to dominate IoT related platforms. That is why we engineered Project Nautilus, a real-time analytics and streaming storage solution, built from the ground up to provide the foundation for reliable streaming applications. This project combines innovative open source streaming software (Pravega), stream analytics (Apache Flink), plus Dell PowerEdge servers which seamlessly tier into our nearly bottomless unstructured data portfolio featuring Isilon & Elastic Cloud Storage (ECS). It’s not available yet—but stay tuned for this to move from project to reality as the world of IoT continues to become increasingly complex.
Check out Project Nautilus here: Dell IQT Keynote.
