Akka, Spark or Kafka? Selecting The Right Streaming Engine For the Job
A Look At Latency, Volume, Integration, And Data Processing Needs
Many engineers we talk to on a daily basis come to us with the same issue: that the batch-oriented architecture of Big Data–where data is captured in large, scalable stores, then processed later–is simply too slow. If you’re looking for a competitive advantage like our clients PayPal, HPE, Starbucks and Capital One, then you’ve got to embrace streaming and “Fast Data” architectures, where data is processed as it arrives.
But for most people we’ve talked to, there is rarely a “one size fits all” technology that can handle all streaming use cases. With so many stream processing tools, which ones should you choose? There are several considerations when making the right selection for the specific needs of your application, such as:
- Latency: How low is necessary? What types of tasks are you processing?
- Volume: How high is required? Is Complex Event Processing involved?
- Integration with other tools: Which ones and how? We know that nothing lives in isolation in these systems
- Data processing: What kinds? In bulk? As individual events?
In this talk by Dean Wampler, PhD., VP of Fast Data Engineering at Lightbend, we’ll look at the criteria you need to consider when selecting technologies, plus the context and background to make good decisions when it comes to adopting streaming frameworks.
Using our Fast Data Platform as an example, which supports a host of Reactive and streaming technologies like Akka Streams, Kafka Streams, Apache Flink, Apache Spark, Mesosphere DC/OS and our own Reactive Platform, we’ll look at how to serve particular needs and use cases in both Fast Data and microservices architectures.
Watch The Full Video (40 Min)
Additional Resources To Enjoy
- An Introduction To Fast Data: Get The Free O'Reilly eBook Fast Data Architectures for Streaming Applications
- Machine Learning In Production: Learn more with Serving Machine Learning Models - A Guide to Architecture, Stream Processing Engines, and Frameworks
- Lightbend Fast Data Platform (PDF): Download our Fast Data Platform technical overview to learn more about our easy-on ramp for designing, building, and running streaming and Fast Data applications.
As always, Lightbend is here to make your streaming, Fast Data, and Machine Learning journey successful. To find out more about our platform subscription, getting-started engagement services, or anything else, feel free to contact us below and schedule a 20-min introduction.