Recently, at Scala Days New York 2016, I had a chance to sit down with Akara Sucharitakul, Principal Member of Technical Staff at PayPal, and the main force behind PayPal's open source squbs project. Akara had recently published an in-depth article about squbs, describing it as the Reactive way to for PayPal to build, deploy and manage crucial services that have to process billions of transactions per day on as little infrastructure as possible.
Large, internet-scale organizations face tremendous challenges building efficient services that would scale and keep a responsive user experience. Services must be able to scale both vertically and horizontally. The programming model should align with open source, keep code and error handling clean and manageable, and retain flexibility to reorganize those services.
squbs (rhymes with "cubes") is an infrastructure for very high transaction density, low latency computing. It allows services to scale vertically to make use of large number of processor cores as well as horizontally to very large number of service instances. The asynchronous programming model allows applications to stay responsive, even under load. It also allows error composition which leads to extremely clean error handling. One core feature of squbs is the flexibility of composing services from loosely-coupled components called "cubes". This allows service boundaries to be adjusted as organizations and business needs change.
"Powered by Akka and Scala, squbs has already provided very high-scale results with a low infrastructure footprint: our applications are able to serve over a billion hits a day with as little as 8 VMs and 2 vCPU each."
squbs is essentially a standardization of Akka for large scale deployments across multiple services allowing hooks for logging, monitoring, security, and other infrastructure needed for running an Internet-scale operation. It rides on Akka and Scala for most of its greatness. Akka's actor model and execution model is the key enabler allowing for high throughput and high transaction density, scaling both vertically and horizontally. Akka's resiliency and supervision allow for the resilience of squbs itself and services built on top of squbs, likewise. The focus on expressiveness and correctness of the Scala language and its libraries allow for writing concise code with concise error handling. The results are impressive. Not only do we achieve much larger transaction rates with the same hardware infrastructure, but we also often cut down code by 80% for the same functionality when compared to equivalent imperative code.
The requirements of highly scalable, responsive, and resilient systems already pointed to the Actor Model of Computation as the potential best-fit. There are only few top contenders in this space: One is Erlang and the other is Akka. PayPal already has a large developer base with Java knowledge and a large investment into Java-based technologies, components, and operationalization. It was natural to consider Akka as the top contender, especially for its Java API support. However, once started, we found the conciseness of Akka's Scala API and the expressiveness and conciseness of Scala in general to be extremely appealing. The core team settled on Scala as their primary language ever since, while still supporting the Java API for any service project that desires to be on Java.
"Akka helps our systems stay responsive even at 90% CPU, very uncharacteristic for our older architectures, and provides for transaction densities never seen before. Batches or micro-batches do their jobs in one-tenth of the time it took before. With wider adoption, we will see this kind of technology being able to reduce cost and allow for much better organizational growth without growing the compute infrastructure accordingly."
While we actively evolve the current squbs core and tooling, we also keep adding more scale-proven programming patterns developers can readily use such as stateful services. Check us out at https://paypal.github.io/squbs and join us on our gitter chat room. Needless to say, help is highly appreciated.
Inspired by this story? Read more!
