Lessons Learned From PayPal: Back Pressure With Akka Streams And Kafka


Lessons Learned From PayPal: Back Pressure With Akka Streams And Kafka

With Akara Sucharitakul

akka-streams reactive-streams

Audience: Architects, Management Technical level: Intermediate

Akka Streams and its amazing handling of streaming with back-pressure should be no surprise to anyone. But it takes a couple of use cases to really see it in action - especially in use cases where the amount of work continues to increase as you’re processing it. This is where back-pressure really shines.

In this talk for Architects and Dev Managers by Akara Sucharitakul, Principal MTS for Global Platform Frameworks at PayPal, Inc., we look at how back-pressure based on Akka Streams and Kafka is being used at PayPal to handle very bursty workloads.

In addition, Akara will also share experiences in creating a platform based on Akka and Akka Streams that currently processes over 1 billion transactions per day (on just 8 VMs), with the aim of helping teams adopt these technologies. In this webinar, you will:

  • Start with a sample web crawler use case to examine what happens when each processing pass expands to a larger and larger workload to process.
  • Review how we use the buffering capabilities in Kafka and the back-pressure with asynchronous processing in Akka Streams to handle such bursts.
  • Look at lessons learned, plus some constructive “rants” about the architectural components, the maturity, or immaturity you’ll expect, and tidbits and open source goodies like memory-mapped stream buffers that can be helpful in other Akka Streams and/or Kafka use cases.


Related Videos

Exploring Reactive Integrations with Akka Streams, Alpakka and Kafka


reactive akka akka-streams

Exploring Reactive Integrations with Akka Streams, Alpakka and Kafka

with Konrad Malawski

Audience: Architects
Technical level: Introductory

Since its stable release in 2016, Akka Streams is quickly becoming the de facto standard integration layer between various Streaming systems and products. Enterprises like PayPal, Intel, Samsung and Norwegian Cruise Lines see this is a game changer in terms of designing Reactive streaming applications by connecting pipelines of back-pressured asynchronous processing stages.

This comes from the Reactive Streams initiative in part, which has been long led by Lightbend and others, allowing multiple streaming libraries to inter-operate between each other in a performant and resilient fashion, providing back-pressure all the way. But perhaps even more so thanks to the various integration drivers that have sprung up in the community and the Akka team—including drivers for Apache Kafka, Apache Cassandra, Streaming HTTP, Websockets and much more.

In this webinar for JVM Architects, Konrad Malawski explores the what and why of Reactive integrations, with examples featuring technologies like Akka Streams, Apache Kafka, and Alpakka, a new community project for building Streaming connectors that seeks to “back-pressurize” traditional Apache Camel endpoints.

  • An overview of Reactive Streams and what it will look like in JDK 9, and the Akka Streams API implementation for Java and Scala.
  • Introduction to Alpakka, a modern, Reactive version of Apache Camel, and its growing community of Streams connectors (e.g. Akka Streams Kafka, MQTT, AMQP, Streaming HTTP/TCP/FileIO and more).
  • How Akka Streams and Akka HTTP work with Websockets, HTTP and TCP, with examples in both in Java and Scala.

Understanding Akka Streams, Back Pressure and Asynchronous Architecture


akka akka-streams reactive-streams reactive

Understanding Akka Streams, Back Pressure and Asynchronous Architecture

with Konrad Malawski

The term 'streams' has been getting pretty overloaded recently–it's hard to know where to best use different technologies with streams in the name. In this talk by noted hAkker Konrad Malawski, we'll disambiguate what streams are and what they aren't, taking a deeper look into Akka Streams (the implementation) and Reactive Streams (the standard).

You'll be introduced to a number of real life scenarios where applying back-pressure helps to keep your systems fast and healthy at the same time. While the focus is mainly on the Akka Streams implementation, the general principles apply to any kind of asynchronous, message-driven architectures.

Audience: Architect
Technical level: Intermediate

  • Report

    Going Reactive 2016

    How Microservices & Fast Data are driving adoption of Reactive systems

    Get Report
  • Video

    How is Typesafe Reactive

    Learn how Typesafe technologies are Reactive

    Watch Now
  • White Paper

    Microservices In Production

    What Ops Should Know About Reactive Systems

    Get White paper