Introducing Lightbend Pipelines: Scalable, Streaming-Data Pipelines On Kubernetes And OpenShift
Lightbend today announced the general availability of Lightbend Pipelines, a new, comprehensive system that accelerates all lifecycle aspects of streaming data pipelines and integrating them with microservices-based applications.
Lightbend Pipelines is a unique, powerful system for building scalable, streaming-data pipelines composed of multi-component streamlets written in one or more streaming engines.
Most streaming data solutions comprise multiple components, implemented in different technologies, which you have to integrate yourself. What you use for streaming ETL is different than what you use for aggregations or model serving. Yet these various components must act as a single service to other application services.
Lightbend Pipelines supports streamlets written in Akka Streams and Spark Structured Streaming, transparently integrated together.
Developer productivity is accelerated by eliminating much of the boilerplate required when writing such applications, especially when composing domain-logic streamlets into complete applications.
The Pipelines runtime environment automates many tasks, such as deployment, routing data between streamlets, and ensuring data safety.
A powerful, intuitive GUI visualizes the running pipelines with easy access to important health and performance metrics.
Management is simplified by automatic handling of many failure scenarios and restarts.
Lightbend Pipelines is available as a new feature of the Lightbend Platform, the most operationally robust solution for deploying integrated Reactive Microservices and streaming data applications. Lightbend Pipelines adds accelerated productivity by eliminating tedious boilerplate during development, simplifying deploying, and providing the runtime visibility you need to ensure application reliability and performance.
To learn more, this 15-minute read describes how Lightbend Pipelines works with microservices to build streaming-data applications. To see Lightbend Pipelines in action, watch this 2-minute video below!