In this "Tales From The Road" story by Michael Read, Senior Consultant at Lightbend, we take a look at achieving operational visibility into the difficult-to-monitor area of asynchronous microservices communication.
The subject of this story is a Lightbend subscriber based in California and specializing in philanthropic projects to support scientific initiatives through research and personalized education.
Specifically, their system uses the Lightbend Platform to process the ever-increasing amount of scientific papers. This includes ingestion, categorization by author and topic(s), scoring by Machine Learning algorithms and indexing. At the end of this, content can be added to a users' feed based on their preferences or subscribed topics.
For the most part, current monitoring philosophy for software is based on decades-old monolithic design, where the database layer, application layer, and front-end web layer are all packaged in a single “box”.
When an error occurs in a monolithic application, the request causing the error runs on a single thread, resulting in a clear call stack from beginning to end. The call stack allows engineers to peek back in time to find the cause of the error.
To contrast, our client’s microservices architecture based on the Akka toolkit and Scala is asynchronous throughout and comprised of a combination of application code, microservices, data frameworks, telemetry, and more. The application code is in many cases also closely intertwined with the foundational data frameworks, making traditional monolith-oriented monitoring tools ineffective at monitoring Reactive applications.
The Lightbend team helped our client increase visibility and and operational monitoring of their existing microservices, including best practices to add custom metrics. This also included enablement and assistance on applying proper Domain-Driven Design (DDD) technique, especially as it applies to distributed and persistent state.
In this engagement, the Lightbend team worked with the client to instrument six microservices in production to utilize custom metrics, with visualizations delivered by Lightbend Telemetry through dashboards for each–an example screenshot of a sample application unrelated to this client is seen above.
Lightbend also helped create a proof of concept–focused on DDD–to track user activity using distributed, scalable, and persistent store of state that could ultimately drive the UI of the client’s website. For later on, we instructed their team on best practices for implementing additional custom metrics as the need arose.
With Lightbend’s expert assistance, our client was able to demystify the black box of asynchronous message passing with Akka Actors for clear visibility into the operation and performance of their existing microservices.
To help continuously improve the client’s user experience, Lightbend delivered a well-developed proof of concept application that would provide ease and speed to their existing UI and could easily be moved into production in short order.
In addition, this relationship provided our customer the opportunity to influence roadmap priorities by working closely with the Lightbend team, including early access to new releases that focused on providing even deeper visibility into Akka Streams with Prometheus.
[Lightbend is looking for people interested in getting to the bottom of tricky challenges like these. Join our team!]
If your team is facing challenges with monitoring streaming, real-time applications and microservices, then Lightbend can help. We offer strategic services, expert-led training courses, and even free online classes (with IBM) on working with Reactive Systems and technologies like Akka, Lagom, Play, Scala, Spark, Kafka, Kubernetes, and more. Contact us to learn how we can help!
