We’re pleased to announce a new release of Lightbend Telemetry, part of Lightbend Platform. Lightbend Telemetry is a suite of observability tools that provides a view into the workings of our distributed platforms built with Akka, Lagom, Play, Java, and Scala. It lets Lightbend customers tap into the “black box” of distributed system tracing by integrating with visualization and monitoring tools like Grafana, New Relic, Datadog, and others.
The Telemetry 2.11 release has focused on covering some of the outstanding context propagation disconnects for Lightbend Platform telemetry (such as Java Futures) and supporting the latest versions of Scala and Java. It includes support for Scala 2.13, telemetry for Java Futures, initial support for Java 11, Akka Scheduler instrumentation, support for log correlation ids, and improvements to various modules. Some deprecated modules have been removed in this release.
Lightbend Telemetry now supports Scala 2.13. Modules and instrumentations are cross-built for Scala 2.13 and the agent can instrument the new Scala 2.13 Future implementation. For the Telemetry 2.11.0 release, support is targeted at Scala 2.13.0-M5 only—the latest release for Scala 2.13 at this time. New Telemetry 2.11.x releases will be available for subsequent Scala 2.13 versions (and is naturally compatible with Scala 2.11 and later).
Telemetry 2.11 now brings equivalent support to Java Futures as exists for Scala Futures. Java CompletableFuture—the default implementation for CompletionStage—is instrumented, and now can record metrics for named CompletableFutures or CompletionStage callbacks, and supports context propagation and tracing.
The Java Future metrics track scheduling, processing, and overall completion for CompletableFutures. This diagram shows the metrics recorded for a thenApplyAsync operation:
The example Grafana dashboards include a dashboard for Java Future metrics:
Named Java Futures will record trace spans when using the OpenTracing integration:
Lightbend Telemetry now includes initial support for running on Java 11. This includes instrumentation for Java 11 CompletableFutures (for a new implementation introduced in Java 9). Note that the Java Module System is not fully supported yet across the complete Lightbend Platform.
Akka Scheduler scheduleOnce is instrumented for context propagation and tracing. This is to support use cases such as triggering timeout messages with scheduleOnce, connecting the context or trace as a single flow. Note that if scheduleOnce is used in a way that forms flow cycles it will create potentially infinite traces—in which case the trace connection can be disabled.
Lightbend Telemetry can add correlation identifiers to the SLF4J MDC for correlating causally related log messages across nodes or threads. Any correlation identifiers will be automatically propagated across asynchronous or remote boundaries with subsequent message sends, HTTP requests, or asynchronous callbacks. You can configure Telemetry to include random correlation IDs automatically, or to include the current trace ID if OpenTracing is being used.
The OpenTracing integration has seen some improvements in Telemetry 2.11. Trace span names and tags can be configured and the defaults are more closely aligned with OpenTracing conventions. Support for debug traces (forced sampling) has been added. Traces that begin with an HTTP request can have a “debug mode” enabled using an HTTP header. When a trace debug header is configured and present on an HTTP request, then the request will have a sampling priority tag added, to hint that this trace should be force sampled. By default, Telemetry will only create trace spans for entities that have been enabled for tracing.
To connect together traces across any intermediary asynchronous boundaries which are not enabled for tracing, Telemetry uses a “pass-through” mode—where the trace context is passed through without any trace spans being created. It’s now also possible to enable “auto tracing” which will automatically create trace spans for the intermediate steps, whenever these occur within a currently active and sampled trace.
Several modules have seen improvements. The StatsD reporters can now report deltas as counters. The Datadog reporter can support the Datadog agent’s Unix Domain Socket mode. PlayWS client requests can be named. Stopwatches are now opt-in (to further reduce the telemetry overhead).
There have been some changes to modules, dependencies, and configuration. The deprecated OverOps and Contrail modules have been removed. The convenience cinnamon-sandbox dependency has been removed (but not the Developer Sandboxes) in favor of using configuration directly. A number of Lightbend Telemetry modules that only contain Java code are no longer cross compiled, and are now published without the trailing Scala version in the artifact name. The OpenTracing backends have been updated. Please take note of the details in the migration guide when upgrading to Telemetry 2.11.
As always, we’re interested in any feedback or ideas for Lightbend Telemetry, to improve the visibility into reactive applications and distributed systems. Reach out to us through Lightbend Support or contact us.
