We are pleased to announce a new release of Lightbend Telemetry, a suite of monitoring and observability tooling for Lightbend technologies. This release features support for recently released Akka 2.6, including support for the new typed Actor APIs (aka Akka Typed) and the Akka Persistence Typed APIs.
The image below provides a small sample of the many metrics Lightbend Telemetry provides for Actors.
Lightbend Telemetry provides deep observability into the inner-workings of distributed applications built with Akka, Lagom, Play, Java, and Scala. This lets users tap into the “black box” of distributed systems via events, metrics and distributed tracing. In addition to Lightbend Console, Telemetry provides a wide variety of integrations, including: Grafana, Prometheus, Elasticsearch, Jaeger, Zipkin, New Relic, Datadog, and others, including any tool that supports OpenTracing.
With Akka 2.6, typed Actors have been marked as “stable”, and now represent the recommended starting point over classic Actors. The typed actor APIs make Akka more approachable, more guided, and provides developers a better experience by catching mistakes earlier when writing applications.
Lightbend Telemetry 2.13 provides complete instrumentation for typed actors, naturally including the same metrics users have enjoyed for classic Actors.
There are some notable differences when it comes to configuration. Unlike classic Actors, all typed Actor instances share the same underlying class, with individual behaviour defined as well-typed behaviors. For this reason, reporting by class is not supported for typed Actors. While reporting by instance and group are fully supported, it is often desirable to group actor metrics with a more specific granularity.
This can be achieved with ActorTags. Introduced in Akka 2.6, `ActorTags` allow arbitrary tags to be assigned to any actors within the system. ActorTags are defined as actor properties(“Props”) at creation time.
For example, if we want to tag all of the “processing” actors, we would create those actors as follows:
context.spawn(heavyBehavior, "HeavyWeightActor", ActorTags("heavyweight"))
Telemetry 2.13 introduces support for including and excluding actors for instrumentation according to their ActorTags, and allows metrics to be reported-by tag, where metrics for actors sharing the same tag are aggregated under it. From the above example, Telemetry may be configured as follows:
cinnamon.akka {
actors {
"tag:heavyweight" {
report-by = tag
}
}
}
Which would provide Actor metrics grouped by all actors with the “heavyweight” tag:
In Lightbend Telemetry 2.12, we introduced metrics and tracing for Akka’s Classic Persistence and Lagom Persistent Entities. With Lightbend Telemetry 2.13, the same metrics and tracing propagation are available for the new Akka Persistence Typed APIs.
These metrics include:
For more details, including examples, see Lightbend Telemetry 2.12 blog post.
Note that as of Lagom 1.6, Akka Persistence Typed has become the recommended persistence API for Lagom event-sourced applications.
Telemetry 2.13 introduces the ability to split HTTP server metrics by status code (e.g. 2xx, 4xx, 5xx, etc). This is useful for separating the metrics of successful requests (e.g. 2xx, etc) from unsuccessful requests (e.g. 4xx, 5xx). This breakdown of metrics is available at both the server level (all endpoints) and the individual endpoint (e.g. “/index.html”) level.
This has been implemented for Play, Lagom and Akka HTTP’s server metrics.
In an effort to simplify the setup of Lightbend’s commercial Libraries, as of Lightbend Telemetry 2.13, the sbt-cinnamon plugin will no longer require the commercial resolver at the project level (plugins.sbt).
Note: The commercial resolver is now required at the project level (build.sbt). A warning will be generated in the logs if this is missing. This has always been the case for Akka Enhancements, hence many users required the commercial resolver in multiple locations. The net impact is that the commercial resolver is only required in one project location.
This release also includes notable performance improvements for Prometheus and Akka Streams users.
For the Prometheus backend, by default, Lightbend Telemetry uses Summaries to capture all time related metrics. Unfortunately, as multiple users have reported, this has resulted in poor performance - significantly worse than other Telemetry backends. As of 2.13, Telemetry uses an HDR Histogram for backing Summary metrics, which results in significant performance improvements when using Telemetry with Prometheus.
For Akka Streams, performance improvements have been made for steams of streams use cases (e.g. streams that product substreams), particularly where streams have been instrumented via configuration.
The above are in addition to other minor fixes and performance improvements.
As always, we’re interested in your feedback or ideas for Lightbend Telemetry, to improve the visibility into Reactive applications and distributed systems. We encourage our clients to reach out to us through the Lightbend Support Portal, or get in touch with your Lightbend representative.
Not a Lightbend client yet? Why not schedule a demo of Lightbend Telemetry with one of our excellent teammates. Let us know when the time is right for a chat:
