Klout’s Big Data Collection Pipelines with Play Framework
January 28, 2013
The fantastic engineering folks at Klout wrote a great blog post last Friday on the usage of Iteratees in Play Framework calculating social influence of users across several social networks.
Our users and clients expect data to be up to date and accurate and it has been a significant technical challenge to reliably meet these goals. In this blog post we describe the usage of Play! Iteratees in our redesigned data collection pipeline. This post is not meant to be a tutorial on the concept of Iteratees, for which there are many great posts already such as James Roper's postand Josh Suereth's post. Rather, this post is a detailed look at how Klout uses Iteratees in the context of large scale data collection and why it is an appropriate and effective programming abstraction for this use case. In a later post we will describe our distributed Akka-based messaging infrastructure, which allowed us to scale and distribute our Iteratee based collectors across clusters of machines.
Head over the the Klout blog to read the rest of the post. Thanks to Naveen Gattu for providing such an excellent post!
You are excited about Reactive applications. And you want to build, manage and monitor them easily and reliably in your environment. We get it. So we built Lightbend Enterprise Suite to help you do that, and more, with a range of powerful Application Management, Intelligent Monitoring, Enterprise Integration and Advanced Tooling features.
One node: Get started by trying the Application Management features on 1 node. Click here to proceed. NO REGISTRATION REQUIRED
Three nodes: Register for free and get started by trying the Application Management features on 3 nodes. Click here to register | Login
More than three nodes: Lightbend customers can use the Application Management features of Enterprise Suite on 3+ nodes. Login here to get started