Big data streaming frameworks today must process immense amounts of data from an expanding set of disparate data sources. Beyond the requirements of fault tolerance, scalability and performance, compelling streaming engines should provide a programming model where computations can be easily expressed and deployed in a distributed manner. Akka is a natural fit to meet these requirements.
GearPump is built on top of Akka. It uses an actor tree to model the whole streaming system. With this hierarchy, many features of Akka such as error supervision, isolation and concurrency etc. are leveraged to build a resilient, scalable and performant system.
With 4 nodes, the whole throughput can reach 11 million messages per second (100 bytes per message) with average latency of 17ms.
Weihua JiangEngineering Manager for BigData Enabling, Intel
Inspired by this story? Contact us to learn more about what Lightbend can do for your organization.
