Lightbend is pleased to announce a new release of the commercial Telemetry module, Cinnamon. The Cinnamon 2.10 release includes Akka Stream telemetry, Scala Future metrics, and improvements to the OpenTracing integration. Cinnamon no longer supports Akka 2.3.
Cinnamon now instruments Akka Streams for metrics and tracing. The Akka Stream metrics include essential metrics for throughput and processing time of stream operators. There are also extended stream metrics for recording end-to-end flow time, stream demand, latency, and per connection metrics. See the documentation for telemetry use cases for Akka Streams that describe how the stream metrics and tracing can be used for performance analysis and illustrate what to expect.
This diagram shows the core stream metrics:
The example Grafana dashboards include dashboards for the core and extended metrics. Here are the core Akka Stream metrics visualized with Grafana:
Cinnamon can now record metrics for named Scala Futures, along with the tracing and context propagation support from earlier Cinnamon versions. The metrics track the scheduling, processing, and overall completion for Futures. This diagram shows the metrics recorded for a Future map operation:
The example Grafana dashboards include a dashboard for Scala Future metrics:
The Cinnamon OpenTracing integration has received some improvements. Cinnamon now supports tracer-specific integration to allow for improved performance — allowing Cinnamon to check details such as the sampling decision for traces, so that extra allocations on non-sampled traces can be avoided as much as possible.
The OpenTracing integration now makes use of trace continuations, where an active span is continued for instrumented but non-traced asynchronous components (such as Scala Futures or Akka Stream operators that are not configured as traceable). Previously, Cinnamon would connect the trace context across these non-traced components, but there wouldn’t be an active span available (for span logging or access to the trace baggage).
Trace continuations also allow Cinnamon to record the total processing time for asynchronous operators in Akka Streams. Asynchronous stream operators, such as mapAsync or mapAsyncUnordered, process elements by registering an asynchronous callback, which will re-enter the stream at a later time once a Future result is ready. This means that the regular processing time and trace spans for these operators will generally be very short, with the bulk of the time spent asynchronously elsewhere. In the trace, this would show as a short span followed by a large gap. With trace continuations, the total time for asynchronous operators will be recorded, including any asynchronous processing or time in buffers, and until a result is pushed downstream. Here’s an example trace that includes a trace continuation span for an asynchronous operator:
Lightbend support for Akka 2.3 expired in April 2016, but Cinnamon has continued to support instrumentation for Akka 2.3. It’s now time to say goodbye to Akka 2.3. Instrumentation for Akka 2.3 was deprecated in Cinnamon 2.9 and has been removed for Cinnamon 2.10. Please contact us through Lightbend Support if you are using Akka 2.3.
As always, we’re interested in any feedback or ideas for Cinnamon, to improve the visibility into reactive applications and distributed systems. Reach out to us through Lightbend Support or contact us.
