Apache Spark

Lightning-fast, distributed compute engine

Apache Spark is a lightning-fast, distributed compute engine for processing large, streaming datasets.

When Spark is combined with your Reactive application, you add sophisticated analytics, including machine learning. You can leverage rich insights from your historical data-at-rest, complemented with real-time updates from your data-in-motion.

Support for Spark is included with an annual subscription to Lightbend Platform.

Lightbend technologies make Spark possible.


and Capabilities

Scala and Java APIs

Lightbend provides support for Spark Core, Spark SQL, Spark Streaming, and Spark MLlib modules, for the Scala and Java APIs, which includes unlimited incidents and answers plus bug fixes and patches.

Spark on Mesos/DCOS

Support covers Mesos/DCOS clustering for the full lifecycle of your project, which means that Spark’s integration with Mesos/DCOS is supported in production environments, but not Mesos itself.

Why Spark and Scala?

Spark is written in Scala, a language that treats data as a first-class citizen. The collections library of the language reflects this—the APIs are designed to provide an easy and standard way to manipulate the data that they contain, based on common constructs that data scientists and developers alike already know.

Why Spark and Mesos/DCOS?

Mesos/DCOS is a next-generation resource manager with the flexibility to manage all applications and services in your entire infrastructure. Separate clusters for Hadoop YARN, Cassandra, etc, are not required, delivering a more efficient approach for companies looking to leverage Spark and other tools for all of their Reactive applications.

Why choose Lightbend?

Hadoop vendors are experienced in supporting production batch jobs in YARN environments. In contrast, our focus is on building streaming applications and ensuring performance is reliable at scale. Because Spark is built on Scala, Lightbend is uniquely qualified to get developers productive quickly when building applications with Spark.

Experts with the Reactive stack

A new constellation of tools is emerging to support fast data applications, which includes Spark, Mesos, Akka, Cassandra, and Kafka (aka the “SMACK stack”). These technologies represent the state-of-the-art in distributed systems and our team at Lightbend has the expertise to understand the roles they play and how you can leverage them in your Reactive systems.