Improved dependency management with sbt 0.13.7

The central features of sbt can be distilled to the following:

• Scala-aware incremental compilation
• User-extensible task/plugin system (using Scala)
• Scala-aware dependency management

As we ramp up for sbt 1.0 in 0.13.5+ technology previews, each of the areas are given attention for enhancements. For instance, the auto plugin system introduced in 0.13.5 significantly reduces the boilerplates associated with introducing plugins to your build; and name hashing that got enabled by default in 0.13.6 significantly improves the efficiency of incremental compilation by tracking the names defined in all compilation units.

In this post, we will go over the dependency management improvements that were added to sbt 0.13.6 and 0.13.7 including:

• Performance improvements with cached resolution
• Eviction warnings
• Unresolved dependency errors

Before we discuss the new cool features, I would like to set up a mental image of how library dependencies looks like inside sbt.

Dependency as a graph

The basics of dependency management is that you specify scalaVersion and a list of libraryDependencies for your project in the form of ModuleID, and sbt's dependency management system downloads the JAR files associated with the direct dependencies as well as the transitive dependencies.

For example, your project may depend on dispatch-core 0.11.2; dispatch-core 0.11.2 depends on async-http-client 1.8.10; async-http-client 1.8.10 depends on netty 3.9.2.Final, and so forth. If we think of each library to be a node with arrows going out to dependent nodes, we can think of the entire list of dependencies to be a graph.

As the number of the nodes increases, the time it takes to resolve dependencies grows significantly (This is caused by sbt dependency resolver resolving version conflicts, evaluating exclusion rules and override rules transitively). The slow resolution is exacerbated for builds with many subprojects as the resolution process is repeated for each subprojects.

Performance improvements with cached resolution

With the sponsorship of LinkedIn's Tools Team, the sbt project has investigated performance improvements on dependency resolution during most of 2014. One of the outcome is an experimental feature added to sbt 0.13.7 called cached resolution.

To enable cached resolution, first set the sbt.version to 0.13.7 in project/build.properties:

sbt.version=0.13.7

Next, enable cached resolution on updateOptions key:

val mahout = "org.apache.mahout" % "mahout-core" % "0.7"
val dispatch = "net.databinder.dispatch" %% "dispatch-core" % "0.11.2"
val scalaCheck = "org.scalacheck" %% "scalacheck" % "1.12.0"

lazy val root = (project in file(".")).
 settings(
   scalaVersion := "2.11.4",
   libraryDependencies ++= Seq(mahout, dispatch, scalaCheck % Test),
   updateOptions :=
updateOptions.value.withCachedResolution(true)
 )

Cached resolution feature is akin to incremental compilation, which only recompiles the sources that have been changed since the last compile. Instead of resolving the full dependency graph, cached resolution feature creates minigraphs for each direct dependency. These minigraphs are resolved using the stock dependency resolver based on Ivy, and the result is stored locally under ~/.sbt/0.13/dependency/ and shared across all builds.

 

 

This allows update task to focus only on resolving newly added dependencies and stitching together all the minigraphs. The process may incur some I/O overhead, but the intended performance improvement is that the second and third subprojects can take advantage of the resolved minigraphs from the first one and avoid duplicated work. Many of the early reports show improvement on update task from 260s to 25s (Your mileage may vary). See the documentation on cached resolution for more details.

Eviction warnings

As the size of the dependency graph grows, another common issue you might encounter is the issue of version conflicts and eviction. A version conflict occurs whenever there are multiple versions suggested for a particular library dependency. For example, your application may depend on slf4j-api 1.6.1 while some other library may transitively depend on 1.7.5.

In the above graph, you can see that both slf4-api 1.6.6 and 1.7.5 appear on the transitive path from root 0.1.0. These version conflicts are resolved by picking the latest version. This process is sometimes called eviction, because from the point of view of 1.6.6, it was evicted in favor of 1.7.5.

Evictions are ok as long as the conflicted versions are binary compatible, like in the case of slf4j-api 1.6.6 and 1.7.5. On the other hand, evictions of binary incompatible library dependencies are quite dangerous. Your code may depend on a public method doSomething from API v1.1 but it might have been removed in API  v2.0. Starting with sbt 0.13.6, sbt will warn you if we suspect evictions of binary incompatible versions. Here's a sample output:

 

[warn] There may be incompatibilities among your library dependencies.
[warn] Here are some of the libraries that were evicted:
[warn]  * com.typesafe.akka:akka-actor_2.10:2.1.4 -> 2.3.7
[warn] Run 'evicted' to see detailed eviction warnings

I was able to produce the warning by depending on banana-rdf 0.4 and Akka Actor 2.3.7. The warning is valid since Akka 2.1.x and 2.3.x are not binary compatible. Here are some of the possible ways to work around this:

See documentation on Library Management for more details on this topic.

Unresolved dependencies error

We have looked into the concept of library dependencies as a graph, and potential eviction issues we can run into as the graph grows. What if the graph cannot be resolved due to missing transitive dependencies?

You might have seen an error message like this:

[warn]     module not found: foundrylogic.vpp#vpp;2.2.1
[warn] ==== local: tried
[warn]   /Users/foo/.ivy2/local/foundrylogic.vpp/vpp/2.2.1/ivys/ivy.xml
[warn] ==== public: tried
[warn]   https://repo1.maven.org/maven2/foundrylogic/vpp/vpp/2.2.1/vpp-2.2.1.pom
[info] Resolving org.fusesource.jansi#jansi;1.4 ...
[warn]     ::::::::::::::::::::::::::::::::::::::::::::::
[warn]     ::          UNRESOLVED DEPENDENCIES         ::
[warn]     ::::::::::::::::::::::::::::::::::::::::::::::
[warn]     :: foundrylogic.vpp#vpp;2.2.1: not found
[warn]     ::::::::::::::::::::::::::::::::::::::::::::::

This tells you what's missing, but often you are left to guess where the transitive dependency came from in the first place. sbt 0.13.6+ will try to reconstruct dependencies tree when it fails to resolve a managed dependency. This is an approximation, but it should help you figure out where the problematic dependency is coming from. When possible sbt will display the source position next to the modules:

[warn]  ::::::::::::::::::::::::::::::::::::::::::::::
[warn]  ::          UNRESOLVED DEPENDENCIES         ::
[warn]  ::::::::::::::::::::::::::::::::::::::::::::::
[warn]  :: foundrylogic.vpp#vpp;2.2.1: not found
[warn]  ::::::::::::::::::::::::::::::::::::::::::::::
[warn]
[warn]  Note: Unresolved dependencies path:
[warn]    foundrylogic.vpp:vpp:2.2.1
[warn]      +- org.apache.cayenne:cayenne-tools:3.0.2
[warn]      +- org.apache.cayenne.plugins:maven-cayenne-plugin:3.0.2 (/foo/build.sbt#L25)
[warn]      +- root:root_2.10:0.1-SNAPSHOT
[trace] Stack trace suppressed: run last *:update for the full output.
[error] (*:update) sbt.ResolveException: unresolved dependency: foundrylogic.vpp#vpp;2.2.1: not found

Now we know that the problematic dependency was maven-cayenne-plugin 3.0.2 included in line 25 of build.sbt. Knowing this, we can either add an appropriate resolver, or exclude the specific transitive dependency from this node. See documentation on Library Management foron how to exclude dependencies.

Summary

Scala-aware dependency management is a core feature of sbt and is based on the concepts of Maven and Apache Ivy. sbt 0.13.6 and 0.13.7 make improvements on dependency resolution for both the performance and user experience.

• Experimental cached resolution feature allows the build to incrementally resolve the dependency graph. This addresses the scalability and performance issues associated with resolving multi-project builds.
• Post-resolution eviction warning feature warns you when binary incompatible evictions are suspected.
• Unresolved dependency error includes the reconstructed dependency tree.

Please try these features from sbt or Activator, and let us know what you think!