New Streaming Data Survey Shows AI, ML and IoT Forcing Real-Time Data Requirements for Developers and Application Stack

BOSTON - RED HAT SUMMIT, May 07, 2019 (GLOBE NEWSWIRE) -- Lightbend today announced the release of a new survey of more than 800 IT professionals who identify as using stream processing in their applications and systems. The 2019 Streaming Data and the Future Tech Stack report conducted by The New Stack --focuses on the use cases, technology choices and obstacles faced by early adopters of real-time data use cases for which streaming data is a major requirement.

The survey found that companies processing data in real-time for AI/ML use cases jumped from 6 percent from 2017 to 33 percent in 2019 -- a more than five-fold increase. IoT experienced a three-fold increase in real-time data processing as another key use case driving streaming adoption.

But while business opportunities for real-time data drive streaming data demand, the survey showed developer experience, familiarity with tools, and technical complexity as barriers to streaming data adoption. The study shows that concern about scalability, latency and other technical challenges increases as the number of workloads utilizing stream processing rises.

We see a renaissance right now where developers are being asked to be a lot more ‘data smart,’ said Mark Brewer, CEO at Lightbend. Streaming data is table stakes for the most interesting future use cases--Artificial Intelligence and Machine Learning most notably--and that’s giving rise to the number of programming languages, frameworks and tools for building and running streaming data-centric applications.

Other highlights from the survey include:

• Organizations that have adopted microservices are the farthest along with stream processing. While 58% of respondents are using microservices in production, that figure jumps to 74% among those with more than a quarter of their applications utilizing stream processing.
• Container orchestration is key to architects’ plans. Fifty-six percent of architects are “extremely likely” to deploy container orchestration within the next 12 months as compared to 42% of all respondents.
• Sixty-eight percent of architects believe it is at least somewhat likely that stream processing will be deployed in the same stack as a container orchestrator like Kubernetes.
• Production-level adoption widened dramatically, with several use cases seeing big jumps over the last two years. The sharp rise in real-time processing for IoT pipelines, ETL and integration of different data streams indicates that organizations need to extract insights from their data and leverage advanced analytics (such as AI/ML) as quickly as possible.

Visit the full survey here, to get in-depth analysis of key sections including:

• Artificial Intelligence and Machine Learning Overtaking Early Adopters’ Use Cases
• Early Adopters Concerned About Unknowns
• Concern About “State” Lessens as More Applications Use Stream Processing
• Kafka Use is Widespread
• Technologists Looking Beyond Kafka for Advanced Use Cases
• Early Use Cases Affect ETL and Messaging Most
• Private Cloud Data Centers Rarely Used in Conjunction with Stream Processing.

As the commercial entity behind the Scala programming language that popular frameworks like Akka, Apache Spark and Apache Kafka are written in, Lightbend is one of the early pioneers behind application infrastructure for some of the industry’s most prominent streaming data success stories. Read streaming data case studies about Lightbend’s work with Capital One, Credit Karma, Hewlett Packard Enterprise, LinkedIn, Verizon and other early streaming adopters.