Last week, the Lightbend team volunteered at the SF Food Bank, and I was utterly surprised to find the experience to be an abject lesson in managing Reactive systems (in production!)
Our task was simple: package rice into one-pound bags for distribution to local families. We started with large, 100-pound bags. We were instructed to divide the work into three steps:
We had a lot of volunteers and many tables, so we distributed into many teams and each team performed these tasks.
We all had a stream of rice coming into our table (in the form of 100-pound bags). The first person performed step 1, filling a plastic bag with approximately one pound of rice, then set the bag on the table and repeated. The next person performed step 2, grabbing a bag of rice then adding or removing rice so the bag contained exactly one pound. The final person performed step 3; sealing, labeling, and boxing each bag of rice. Since we were working both sides of the tables, each table had 6 people with each side performing the 3 steps.
Here’s a picture of a table of Lightbenders packaging rice:
In the picture above, you can see Seth at the far end of the table scooping large amounts of rice into a bin. Then Mark and Richard perform step 1, Sushila and Garth perform step 2, and finally Shobhit and Brad complete step 3 (note that Lightbenders insist on doing everything in style, thus the attractive hairnets).
I was impressed by this table of Lightbenders, because without any special plan in mind they went about applying the concepts of microservices, actors, and Reactive Streams to optimize their process. Here’s how:
They noticed that step 3 was slowing down their system. There was too much to do. Step 3 was actually three mini-jobs (let’s call them microservices why not?): (1) sealing, (2) labeling, and (3) boxing.
They redefined their system so that each of these were discrete activities. They defined each service to do one thing very well, rather than keeping the larger combined service that was the original step 3. Each service was just the right size. To use the Swedish word, each service was lagom.
They also realized that each person on the team was capable of performing any one of the tasks. Each person was an actor in the system, and so could accept any message. If someone was idle, they would move to another step and perform the new task. If there was a system failure (for example, a faulty scale), the supervisor brought a new scale, and the team moved people to around to ensure they were being used most effectively.
Above, you can see the Lightbend team with Garth performing step 1, Mark and Sushila performing step 2, Brad and Richard sealing the bags, while Shobhit labeled and boxed the bags.
Finally, they used the back-pressure mechanism of Reactive Streams to introduce the concept of demand flow to their system. A demand flow sets up a line of communication that ensures that there will never be a case where someone needs to be told to slow down. For example, if the people at step 1 produce too many bags of rice, the people at step 2 would tell them to slow down, as well as recommending the velocity of incoming bags to make sure demand and supply are evenly paced.
My table was not as successful. We encountered the exact problem that back-pressure would have solved, with a table overflowing with ~1 pound bags of rice. We nearly experienced catastrophic failure, with unsealed bags ready to fall on the floor.
In the image above, you can see that my table lacked an agreed upon protocol for back-pressure: bags of rice piled between step 1 and step 2, and left no remaining space on the table for more bags. Due to language challenges our communications were limited to smiling. I was assigned to step 1. When the pile got so high that an unsealed bag almost fell on the floor, I decided I would temporarily switch tables. I joined my fellow Lightbenders, adding a 7th person to their table.
This was my final lesson from the experience: a well designed microservice system uses resources more efficiently.
By applying the concepts of microservices, actors, and back-pressure the Lightbend table could use more resources to accelerate the process and deliver far more. Had I remained idle on my original table, we would have been wasting cycles to do work.
I expected that I would value the experience; giving a small amount back to our local community and doing something quite different from our normal work day. But I did not, however, expect to find that we’d apply many of the concept that Lightbenders use every day to build fantastic products.
Author's note: I avoided any photos of myself wearing a hairnet. I’m told I look like Norman Bates’ mother working a food line.
If you want to learn more about these topics, Lightbend has what you need. With so much activity in this area, the term 'streams' has been getting pretty overloaded recently–it's hard to know where to best use different technologies with streams in the name.
In Understanding Akka Streams, Back Pressure and Asynchronous Architectures, noted hAkker Konrad Malawski disambiguates what streams are and what they aren't, taking a deeper look into Akka Streams (the implementation) and Reactive Streams (the standard).
