Lightbend Telemetry 2.15 Released

The Telemetry team at Lightbend is pleased to announce a new release of Lightbend Telemetry, a suite of monitoring and observability tooling for Lightbend technologies. This release brings some new capabilities to Telemetry, including:

• Support for Akka Projections, including metrics for active projections, waiting time, and events for processing errors.
• Improved documentation with a new theme, metrics dictionary, search, and comprehensive sample projects:
  • The new documentation theme is consistent with docs for other products, and easier to navigate.
  • We also now have a metrics dictionary where you can easily check all the metrics and events available when using Lightbend Telemetry.
  • You can now easily search through the documentation too.
  • We added comprehensive samples for Akka and Lagom, using Scala, Java, Maven, and sbt. The samples are based on the new Akka Platform Guide, and Lagom Shopping-Cart.
• We are dropping support for Play 2.6 since it was EOLed on Feb 1, 2020. Lightbend Telemetry officially supports Play 2.8 from version 2.14.

Lightbend Telemetry provides deep observability into the inner-workings of distributed applications built with Akka, Lagom, Play, Java, and Scala. This lets users tap into the “black box” of distributed systems via events, metrics and distributed tracing. Telemetry provides a wide variety of integrations, including: Grafana, Prometheus, Elasticsearch, Jaeger, Zipkin, New Relic, Datadog, and others, including any tool that supports OpenTracing.

FREE COURSE: Monitoring Akka with Lightbend Telemetry

Akka Projection Metrics and Tracing

Telemetry 2.15 introduces support for instrumenting services that use Akka Projections to process a stream of events or records from a source to a projected model or external system. The screenshots below show the Prometheus/Grafana-backed dashboard alongside an explanation of the metrics included:

Multiple envelope metrics are available, such as:

• Offset Success - number of envelopes marked as processed or number of offset positions advanced
• Handler Success - number of processed envelopes or number of times an envelope handler ran successfully
• Handler Error - error rate during the envelope processing–if you use a handler strategy including a retry operation, this number will increase

You can mouse over to see detailed information per projection:

In combination with envelope metrics and events, there are metrics specific to projections themselves:

Here you can see metrics such as:

• Projection Count - The number of running projections and a counter of instances, so if you run a sharded projection the counter will indicate the number of shards
• Projection Failure - The failure rate for projections, where failures indicate that the projection required restarting
• Residence Time - The time (in milliseconds) spent from the creation of the envelope to when its offset was committed
• Service Time - The time (in nanoseconds) from when the envelope was considered ready until the projected data was available for reading
• Wait time - The time (in milliseconds) an envelope spent on the queue/broker/journal–also the time since the envelope was created until it was read, deserialized, and considered ready to be processed by the projection.

See more details about these metrics and others in our new metrics dictionary.

Improvements in Datadog tracing compatibility

There were some cases where Datadog tracing spans were not connecting properly when the application is instrumented by both cinnamon-agent and datadog-agent. The main cause was the currently active context not being captured as expected, or Lightbend Telemetry expecting a proper implementation of equality comparison for spans.

Both agents also try to create a global tracer, leading to the application having multiple global tracers. Moreover, both agents try to instrument the same code (Akka dispatcher executor), so one overrides another. Although it is unusual to have the two agents running side by side, fixing these problems makes the result more predictable.

Fixes, Enhancements and Performance Improvements

This release also contains a number of fixes, enhancements and performance improvements that are fully documented in the release notes with links through to the relevant documentation where appropriate.

Connect with Lightbend

As always, we’re interested in your feedback or ideas for Lightbend Telemetry, to improve the visibility into Reactive applications and distributed systems. We encourage our clients to reach out to us through the Lightbend Support Portal, or get in touch with your Lightbend representative.

To get a first-hand experience with Telemetry, visit the free Lightbend Academy course "Monitoring Akka with Lightbend Telemetry":

VISIT THE ACADEMY COURSE