New Name - Cloudflow is now Akka Data Pipelines
Lightbend recently introduced Cloudflow, an open source framework hosted at cloudflow.io that addresses the full application lifecycle for developing, deploying and operating streaming data pipelines on Kubernetes.
Cloudflow enables users to quickly develop, orchestrate, and operate distributed streaming applications on Kubernetes. Cloudflow allows you to easily break down your streaming application into smaller components and wire them together with schema-based contracts. Cloudflow integrates with popular streaming engines like Akka Streams, Apache Spark, and Apache Flink. It also comes with a comprehensive CLI tool to easily manage, scale and configure your streaming applications at runtime. With its powerful abstractions, Cloudflow allows you to define, build, deploy, and evolve the most complex streaming applications.
As data pipelines become first-class citizens in microservices architectures, Cloudflow gives developers data-optimized programming abstractions and run-time tooling for Kubernetes. In a nutshell, Cloudflow is an application development toolkit comprising:
Streamlet, the core abstraction in Cloudflow.
Streamlet(s). Cloudflow today provides support for popular streaming runtimes, like Spark Structured Streaming, Flink, and Akka Streams.
Streamletcomposition model driven by a
sbtplugins that package your application into a deployable container.
kubectlplugin, that facilitates manual and scripted management of the application.
The different parts of Cloudflow work in unison to dramatically accelerate your application development efforts, reducing the time required to create, package, and deploy an application from weeks to hours.
To read more about Cloudflow concepts, check out our documentation.
Cloudflow’s source code is available on GitHub at https://github.com/lightbend/cloudflow.
The project documentation contains all you need to know to get started with Cloudflow. Developers can get started in one of two ways:
Regardless of which approach you use, you can start with our sample application, which simulates processing data from a wind turbine farm.
Lightbend offers commercial features for Cloudflow that help you productionize and operate the full application lifecycle of a Cloudflow application.
Our UI provides an easily comprehended picture of end-to-end health and performance of your application. The UI makes it easy to quickly pinpoint unhealthy streamlets, allowing you to visualize processing bottlenecks. Context-sensitive charts allow you to drill into specifics of problematic streamlets.
The production tooling provides additional Kubernetes operators to augment the operational experience of Cloudflow.
A subscription to Lightbend Platform comes with these features, full support, and, of course, the right to use the rest of the Lightbend Platform.
We welcome discussion, questions, and contributions. There are two ways to interact with our community.
Please use whichever forum you find most convenient and appropriate.
Please check our Code of Conduct. Be kind, courteous, and empathetic and you’ll be good!