Hi everyone. On behalf of the sbt project, I am happy to announce sbt 1.3.0-RC1. This is the third feature release of sbt 1.x, a binary compatible release focusing on new features. sbt 1.x is released under Semantic Versioning, and the plugins are expected to work throughout the 1.x series.

  • If no serious issues are found by Wednesday, May 29th 2019, 1.3.0-RC1 will become 1.3.0 final.

The headline features of sbt 1.3 are out-of-box Coursier library management, ClassLoader layering, IO improvements, and super shell. Combined together we hope these features will improve the user experience of running your builds.

Changes with compatibility implication

  • Library management with Coursier. See below for details.
  • ClassLoader layering. See below for details.
  • Super shell. See below for details.
  • Multi command no longer requires leading semicolon. clean;Test/compile; would work. #4456 by @eatkins
  • Some functions under sbt.internal.inc.ZincUtil that uses LM have moved to ZincLmUtil to break LM dependency from Zinc. zinc#655 by @dwijnand

Library management with Coursier

sbt 1.3.0 adopts Coursier for the library management. Coursier is a dependency resolver like Ivy, rewritten in Scala by Alexandre Archambault (@alexarchambault), aiming to be a faster alternative.

Note: Under some situations, Coursier may not resolve the same way as Ivy (for example remote -SNAPSHOTs are cached for 24 hours). If you wish to go back to Apache Ivy for library management, put the following in your build.sbt:

ThisBuild / useCoursier := false

Many people were involved in the effort of bringing Coursier to sbt. Early in 2018 Leonard Ehrenfried (@leonardehrenfried) started the Coursier-backed LM API implementation as lm#190. During the fall, it was further improved by Andrea Peruffo (@andreaTP), and lm-coursier eventually became part of coursier/sbt-coursier repository maintained by Alex. This spring, Eugene (@eed3si9n) revisited this again to make a few more changes so we can swap out the LM engine in #4614 with the help from Alex.

ClassLoader layering

sbt has always created two-layer ClassLoaders when evaluating the run and test tasks. The top layer of the ClassLoader contains the scala library jars so that the classes in the scala package may be reused across multiple task evaluations. The second layer loads the rest of the project classpath including the library dependencies and project class files.

sbt 1.3.0 will default to using a three-layer ClassLoader where the second layer can load the library dependency jars and the third layer can load the project jars and class files (ClassLoaderLayeringStrategy.TestDependencies for Test configuration and ClassLoaderLayeringStrategy.ScalaLibrary for others). By caching the library jar classloader, the startup latency of the run and test tasks can be reduced significantly when they are run multiple times within the same session. GC pressure is also reduced because libraries jars will not be reloaded every time the task is evaluated.

Note: This change may affect Java serialization if a library attempts to serialize or deserialize classes that are in the project classpath. In those cases, you can opt out of layering altogether by picking ClassLoaderLayeringStrategy.Flat. See #89.

sbt 1.3.0 introduces classLoaderLayeringStrategy setting that furthers this concept.

Compile / classLoaderLayeringStrategy := ClassLoaderLayeringStrategy.Flat
Compile / classLoaderLayeringStrategy := ClassLoaderLayeringStrategy.ScalaLibrary
Compile / classLoaderLayeringStrategy := ClassLoaderLayeringStrategy.RuntimeDependencies

Test / classLoaderLayeringStrategy := ClassLoaderLayeringStrategy.Flat
Test / classLoaderLayeringStrategy := ClassLoaderLayeringStrategy.ScalaLibrary
Test / classLoaderLayeringStrategy := ClassLoaderLayeringStrategy.RuntimeDependencies
Test / classLoaderLayeringStrategy := ClassLoaderLayeringStrategy.TestDependencies
Test / classLoaderLayeringStrategy := ClassLoaderLayeringStrategy.ShareRuntimeDependenciesLayerWithTestDependencies
  • ClassLoaderLayeringStrategy.Flat includes all classes and JARs except for the Java runtime. The behavior of tasks using this strategy should be similar to forking without the overhead of starting a new jvm.
  • ClassLoaderLayeringStrategy.ScalaLibrary creates a two-layer ClassLoader where Scala standard library is kept warm
  • ClassLoaderLayeringStrategy.RuntimeDependencies creates a three-layer ClassLoader where runtime dependencies are kept warm
  • ClassLoaderLayeringStrategy.TestDependencies creates a three-layer ClassLoader where test dependencies are kept warm
  • ClassLoaderLayeringStrategy.ShareRuntimeDependenciesLayerWithTestDependencies creates a four-layered ClassLoader where the test dependencies layer is added on top of the runtime dependencies layer. As the name implies, the first two layers are shared with the runtime layer so that compile/runtime dependency libraries are only ever loaded once. Note: This might become inconsistent in case test dependencies are supposed to evict some runtime dependencies.

ClassLoaderLayeringStrategy.RuntimeDependencies and ClassLoaderLayeringStrategy.TestDependencies should benefit the response time of run and test tasks, and reduce the GC pressure.

ClassLoaderLayeringStrategy.Flat on the other hand will benefit certain applications that do not work well with layered ClassLoaders. One such example is Java serialization + serialization proxy pattern used by Scala collections.

ClassLoader layering was contributed by Ethan Atkins (@eatkins) as #4476

IO improvements

In addition to classloader layering, sbt 1.3.0 incorporates numerous performance enhancements including:

  • faster recursive directory listing -- sbt internally uses a native library, swoval, that provides a jni interface to native os apis that allow for faster recursive directory listing than the implementations in the java standard library.
  • reduced latency of file change detection in continuous builds. In most cases file events will trigger task evaluation within 10ms.

As of this writing sbt 1.3.0's edit-compile-test loop for 5000 source files is faster than that edit-compile-test with three source files using sbt 0.13, Gradle, and other build tools we tested (See build performance for details). These changes were contributed by Ethan Atkins (@eatkins).

Glob

sbt 1.3.0 introduces a new type, Glob, that describes a path search query similar to Unix shell globs. For example, all of the scala sources in the project directory can be described by Glob(baseDirectory.value, RecursiveGlob / "*.scala") or baseDirectory.value.toGlob / ** / "*.scala", where ** is an alias for RecursiveGlob. Glob expands on PathFinders but they can be composed with no io overhead. Globs can be retrieved using a FileTreeView. For example, one can write:

val scalaSources = baseDirectory.value.toGlob / ** / "*.scala"
val javaSources = baseDirectory.value.toGlob / ** / "*.java"
val allSources = fileTreeView.value.list(Seq(scalaSources, javaSources))

and the FileTreeView will only traverse the base directory once. Globs and FileTreeView were added by Ethan Atkins (@eatkins) in io#178io#216, and io#226

Watch improvements

sbt 1.3.0 introduces a new file monitoring implementation. It uses enhanced APIs for tracking file change events using os events. It adds a new parser that extracts the specific task(s) for which it will monitor source files and rerun when it detects changes. Only source dependencies of the running tasks are monitored. For example, when running ~compile, changes to test source files will not trigger a new build. Between file events, there are also now options to return to the shell, rerun the previous command(s) or exit sbt. These changes were implemented by Ethan Atkins (@eatkins) in io#178#216#226#4512, and #4627.

Build definition source watch

sbt 1.3.0 automatically watches the build definition sources and displays a warning if you execute a task without reloading. This can be configured to reload automatically as follows:

Global / onChangedBuildSource := ReloadOnSourceChanges

This feature was contributed by Ethan Atkins (@eatkins) in #4664

Custom incremental tasks

sbt 1.3.0 provides support to implement custom incremental tasks based on files. Given a custom task that returns java.nio.file.Path, Seq[java.nio.file.Path], File, or Seq[File], you can define a few helper tasks to make it more incremental.

import java.nio.Path
val gccCompile = taskKey[Seq[Path]]("compile C code using gcc")
val gccLink = taskKey[Path]("link C code using gcc")

gccCompile / sourceDirectory := sourceDirectory.value
gccCompile / fileInputs := {
  val base: Glob = (gccCompile / sourceDirectory).value.toGlob
  base / ** / "*.c" :: base / "include" / "*.h" :: Nil
}
gccCompile / target := baseDirectory.value / "out" / "objects"

gccCompile := {
  gccCompile.previous match {
    val changedFiles: Option[Seq[Path]] = (gccCompile / changedInputFiles).value match {
      case Some(ChangedFiles(c, _, u)) => Some(c ++ u)
      case None => None
    }
    case Some(outputs: Seq[Path]) if changedFiles.isEmpty =>
      outputs
    case _ =>
      // do something and generate files in (gccCompile / target)
  }
}

Given these setup, gccCompile / allInputFiles will return sequence of all input files, gccCompile / changedInputFiles returns only the changed files since the last run. These can be used together with gccCompile.previous value feature to avoid unnecessary processing.

In another task such as gccLink, the result of gccCompile can be tracked as well using gccCompile / changedOutputFiles.

gccLink := {
  val changedObjs = (gccCompile / changedOutputFiles).value
  gccLink.previous match {
    case Some(p: Path) if changedObjs.isEmpty =>
      p
    case _ =>
      // do something
  }
}

The inputs of a task will automatically be monitored by the command which has a new parser that is context aware. A custom clean task is also implemented for any task that generates file outputs. The clean tasks are aggregated across the project and config scopes. For example, Test / clean will clean all of the files generated by tasks in the Test config declared in the Test config but not the files generated in the Compile config.

This feature was contributed by Ethan Atkins (@eatkins) in #4627.

Super shell

When running in an ANSI-compatible terminal, sbt 1.3.0 will display the currently running tasks. This gives the developer the idea of what tasks are being processed in parallel, and where the build is spending its time. In homage to Gradle's "Rich Console" and Buck's "Super Console", we call ours "Super shell."

To opt-out put the following in the build:

ThisBuild / useSuperShell := false

or run sbt with --supershell=false (or -Dsbt.supershell=false). This feature was added by Eugene Yokota (@eed3si9n) as #4396/util#196.

Tracing

To view the task breakdown visually, run sbt with --traces (or -Dsbt.traces=true). This will generate build.traces file, which is viewable using Chrome Tracing chrome://tracing/. This feature was contributed by Jason Zaugg (@retronym).

To output the task timings on screen, run sbt with --timings (or -Dsbt.task.timings=true -Dsbt.task.timings.on.shutdown=true).

SemanticDB support

sbt 1.3.0 makes it easier to generate SemanticDB. To enable the generation of SemanticDB build-wide:

ThisBuild / semanticdbEnabled := true
ThisBuild / semanticdbVersion := "4.1.9"
ThisBuild / semanticdbIncludeInJar := false

This was added by @eed3si9n as #4410.

print command

sbt 1.3.0 adds a new print command, similar to show but prints directly to standard out.

# sbt -no-colors --error  "print akka-cluster/scalaVersion"
2.12.8

This was contributed by David Knapp (@Falmarri) as #4341

Appending Function1

Function1 can be appened using +=.

Global / onLoad += { s =>
  doSomething()
  s
}

This was contributed by Dale Wijnand (@dwijnand) as #4521.

JDK 11 support

sbt 1.3.0 is first release of sbt that's been testing on JDK11 extensively.
All integration tests on Travis CI are on AdoptOpenJDK's JDK 11, which were updated by @eed3si9n as #4389/zinc#639/zinc640.

  • Fixes warnings on JDK 9+ by upgrading to protobuf 3.7.0 zinc#644 by @smarter
  • Fixes spurious rebuilds caused by invalidation of rt.jar on JDK 11 #4679 by @eatkins

 

Other bug fixes and improvements

  • Fixes cross building with a single-letter alias #4355 / #1074 by @eed3si9n
  • Removes old warning about global directory #4356 / #1054 by @eed3si9n
  • Improves JDK discovery for cross-JDK forking #4313 / #4462 by @raboof
  • Expands ~ in -Dsbt.global.base property to user home. #4367 by @kai-chi
  • Adds def sequential[A](tasks: Seq[Initialize[Task[A]]]): Initialize[Task[A]]. #4369 by @3tty0n
  • Fixes sbt server to send error event on command failure. #4378 by @andreaTP
  • Implements cancellation of request by LSP client. #4384 by @andreaTP
  • Implements "sbt/completion" command in sbt to server to complete sbt commands. #4397 by @andreaTP
  • Fixes errors order reported by sbt server. #4497 by @tdroxler
  • Fixes cached resolution. #4424 by @eed3si9n
  • The sbt task definition linter warns rather than errors by default. The linter can be disabled entirely by putting import sbt.dsl.LinterLevel.Ignore in scope. #4485 by @eatkins
  • Full GC is only automatically triggered when sbt has been idle for at least a minute and is only run at most once between shell commands. This improves shell responsiveness. #4544 by @eatkins
  • Avoids NPE in JDK12. #4549 by @retronym
  • Fixes the eviction warning summary lm#288 by @bigwheel
  • Fixes Zinc's flag to skip the persistence of API info. zinc#399 by @romanowski
  • Fixes Zinc not detecting synthetic top level member changes. #4316/zinc#572 by @jvican
  • Zinc to notify callback of generated non-local classes before the compiler's middle and backend phases. zinc#582 by @jvican
  • Removes a use of regex in Zinc for performance. zinc#583 by @retronym
  • Fixes incremental compilation involving default arguments. zinc#591 by @jvican
  • Adds Analysis callback of Zinc thread-safe. zinc#626 by @dotta
  • Fixes a non-zero exit Javadoc not failing the task. zinc#625 by @raboof

Participation

First, I'd like to introduce Ethan Atkins, a core community member of the sbt project, and author of Close Watch, which uses native code to provide watch service on macOS. Normally I don't publicize the number of commits, but here's the top 10 for sbt 1.3.0:

323 Ethan Atkins
261 Eugene Yokota (eed3si9n)
42  Jorge Vicente Cantero (jvican)
35  Łukasz Wawrzyk
28  Dale Wijnand
24  Andrea Peruffo
11  Guillaume Martres
7   Jason Zaugg
7   Kenji Yoshida (xuwei-k)
6   Arnout Engelen

As a community member, Ethan has contributed various IO related improvements to make sbt more responsive in his own time. sbt 1.3.0 reflects many of his ideas.

The last feature release of sbt 1 was sbt 1.2.0 in July, 2018. Since then, we've released eight patch releases under sbt 1.2.x for bug fixes, but most of the feature enhancements were merged to develop branch. Over the course of these months, 38 contributors contributors participated in sbt 1.3.0 and Zinc: Ethan Atkins, Eugene Yokota (eed3si9n), Jorge Vicente Cantero (jvican), Łukasz Wawrzyk, Dale Wijnand, Andrea Peruffo, Guillaume Martres, Jason Zaugg, Kenji Yoshida (xuwei-k), Arnout Engelen, Krzysztof Romanowski, Antonio Cunei, Mirco Dotta, OlegYch, Nepomuk Seiler, 0lejk4, Alex Dupre, Alexandre Archambault, Eric Peters, Kazuhiro Sera, Philippus, Som Snytt, Thomas Droxler, Veera Venky, bigwheel, Eugene Platonov, Helena Edelson, Ignasi Marimon-Clos, Julien Sirocchi, Justin Kaeser, Kajetan Maliszewski, Leonard Ehrenfried, Ólafur Páll Geirsson, Stefan Wachter, Yusuke Izawa, falmarri, kai-chi, tanishiking. Thank you!

Thanks to everyone who's helped improve sbt and Zinc 1 by using them, reporting bugs, improving our documentation, porting builds, porting plugins, and submitting and reviewing pull requests. For anyone interested in helping sbt, there are many avenues for you to help, depending on your interest. If you're interested, Contributing, sbt-contrib, "help wanted", "good first issue" are good starting points.

Share



Comments


View All Posts or Filter By Tag


Questions?