Lessons Learned From PayPal: Implementing Back-Pressure With Akka Streams And Kafka

Technologies PayPal Uses To Process 1 Billion Transactions / Day

View on Slideshare

I first met Akara Sucharitakul, Principal MTS for Global Platform Frameworks at PayPal at Scala Days 2016 in New York. He was taking booth duty while his team attended sessions, and we got talking about the squbs project, PayPal's OSS Reactive streaming platform based on Akka.

I was shocked to hear Akara’s description of squbs performance of processing over 1 billion transactions a day on just 8 VMs using Akka cluster and other parts to the Akka toolkit (see my subsequent blog post).

So, I was excited to ask Akara to join our webinar program to talk a bit more on how back-pressure based on Akka Streams and Kafka is being used in PayPal's mission-critical production systems to handle very bursty workloads...

About This Presentation

Akka Streams and its amazing handling of streaming with back-pressure should be no surprise to anyone. But it takes a couple of use cases to really see it in action - especially in use cases where the amount of work continues to increase as you’re processing it. This is where back-pressure really shines.

In addition, Akara will also share experiences in creating a platform based on Akka, Akka Streams, Kafka, Scala and other technologies, with the aim of helping teams see why it's sensible to adopt these technologies. In this presentation, Akara goes into detail on:

• PayPal's new Reactive web crawler based on Akka in a use case to examine what happens when each processing pass expands to a larger and larger workload to process.
• How PayPal uses the buffering capabilities in Kafka and the back-pressure with asynchronous processing in Akka Streams to handle such bursts.
• Lessons learned, plus some constructive “rants” about the architectural components, the maturity, or immaturity you’ll expect, and tidbits and open source goodies like memory-mapped stream buffers that can be helpful in other Akka Streams and/or Kafka use cases.
• BONUS: Some opinions and answers in an extended Q/A, including how PayPal rebuilt their legacy Spring web crawler with Akka to gain a 10x performance increase on less infrastructure.

---

Watch The Full Presentation (45 Min + Q/A)

Watch on YouTube

---

Check Out These Additional Resources

• RESOURCE PACK: 7 Lightbend resources to help you get started with streaming systems using Akka, Kafka, Spark, Cassandra, Mesos and other technologies
• WATCH: Understanding Akka Streams, Back-Pressure and Async Architectures, by Konrad Malawski
• O'REILLY EBOOK: Designing Reactive Systems: The Role Of Actors In Distributed Architecture, by Hugh McKee 

Have more questions? Want to talk to someone at Lightbend? Find out more about how Lightbend can help your team accelerate your modernization initiatives with a 15-20 chat:

GET IN TOUCH WITH US