Hi everyone. On behalf of the sbt project, I am happy to announce sbt 1.3.0. This is the third feature release of sbt 1.x, a binary compatible release focusing on new features. sbt 1.x is released under Semantic Versioning, and the plugins are expected to work throughout the 1.x series.

The headline features of sbt 1.3 are out-of-box Coursier library management, IO improvements, super shell, and experimental ClassLoader layering. Combined together we hope these features will improve the user experience of running your builds.

How to upgrade

You can upgrade to sbt 1.3.0 by putting the following in project/build.properties:

sbt.version=1.3.0

To get other benefits to launching script and sbt launcher, we strongly recommend upgrading the sbt script following the setup instruction.

Changes with compatibility implication

  • Library management with Coursier. See below for details.
  • Super shell. See below for details.
  • Multi command no longer requires leading semicolon. clean;Test/compile; would work. #4456 by @eatkins
  • Deprecates HTTP resolvers, but allow localhost or resolvers marked .withAllowInsecureProtocol(true) #4997
  • Deprecates CrossVersion.Disabled. Please use CrossVersion.disabled instead sbt/librarymanagement#316
  • ClassLoader management: To prevent resource leaks, sbt 1.3.0 closes the ephemeral ClassLoaders used by the run and test tasks after those tasks complete. This may cause downstream crashes if the task uses ShutdownHooks or if any threads created by the tasks continue running after the task completes. To disable this behavior, either set Compile / run / fork := true or run sbt with -Dsbt.classloader.close=false.

Library management with Coursier

sbt 1.3.0 adopts Coursier for the library management. Coursier is a dependency resolver like Ivy, rewritten in Scala by Alexandre Archambault (@alexarchambault), aiming to be a faster alternative.

Note: Under some situations, Coursier may not resolve the same way as Ivy (for example remote -SNAPSHOTs are cached for 24 hours). If you wish to go back to Apache Ivy for library management, put the following in your build.sbt:

ThisBuild / useCoursier := false

Many people were involved in the effort of bringing Coursier to sbt. Early in 2018 Leonard Ehrenfried (@leonardehrenfried) started the Coursier-backed LM API implementation as lm#190. During the fall, it was further improved by Andrea Peruffo (@andreaTP), and lm-coursier eventually became part of coursier/sbt-coursier repository maintained by Alex. This spring, Eugene (@eed3si9n) revisited this again to make a few more changes so we can swap out the LM engine in #4614 with the help from Alex.

Turbo mode with ClassLoader layering

sbt 1.3.0 adds "turbo" mode that enables experimental or advanced features that might require some debugging by the build user when it doesn't work.

ThisBuild / turbo := true

Initially we are putting the layered ClassLoader (ClassLoaderLayeringStrategy.AllLibraryJars) behind this flag.

sbt has always created two-layer ClassLoaders when evaluating the run and test tasks. The top layer of the ClassLoader contains the scala library jars so that the classes in the scala package may be reused across multiple task evaluations. The second layer loads the rest of the project classpath including the library dependencies and project class files. sbt 1.3.0 introduces experimental classLoaderLayeringStrategy feature that furthers this concept.

Compile / classLoaderLayeringStrategy := ClassLoaderLayeringStrategy.Flat
// default
Compile / classLoaderLayeringStrategy := ClassLoaderLayeringStrategy.ScalaLibrary
// enabled with turbo
Compile / classLoaderLayeringStrategy := ClassLoaderLayeringStrategy.AllLibraryJars

Test / classLoaderLayeringStrategy := ClassLoaderLayeringStrategy.Flat
// default
Test / classLoaderLayeringStrategy := ClassLoaderLayeringStrategy.ScalaLibrary
// enabled with turbo
Test / classLoaderLayeringStrategy := ClassLoaderLayeringStrategy.AllLibraryJars
  • ClassLoaderLayeringStrategy.Flat includes all classes and JARs except for the Java runtime. The behavior of tasks using this strategy should be similar to forking without the overhead of starting a new jvm.
  • ClassLoaderLayeringStrategy.ScalaLibrary creates a two-layer ClassLoader where Scala standard library is kept warm, similar to sbt 1.2.x
  • ClassLoaderLayeringStrategy.AllLibraryJars creates a three-layer ClassLoader where library dependencies, in addition to Scala standard libraries are kept warm

ClassLoaderLayeringStrategy.AllLibraryJars should benefit the response time of run and test tasks. By caching the library jar classloader, the startup latency of the run and test tasks can be reduced significantly when they are run multiple times within the same session. GC pressure is also reduced because libraries jars will not be reloaded every time the task is evaluated.

Note: ClassLoaderLayeringStrategy.AllLibraryJars reuses the singleton object between the tests, which requires libraries to clean after itself.

ClassLoaderLayeringStrategy.Flat on the other hand will benefit certain applications that do not work well with layered ClassLoaders. One such example is Java serialization + serialization proxy pattern used by Scala collections.

ClassLoader layering was contributed by Ethan Atkins (@eatkins) as #4476

IO improvements

In addition to classloader layering, sbt 1.3.0 incorporates numerous performance enhancements including:

  • faster recursive directory listing -- sbt internally uses a native library, swoval, that provides a jni interface to native os apis that allow for faster recursive directory listing than the implementations in the java standard library.
  • reduced latency of file change detection in continuous builds. In most cases file events will trigger task evaluation within 10ms.

As of this writing sbt 1.3.0's edit-compile-test loop for 5000 source files is faster than that edit-compile-test with three source files using sbt 0.13, Gradle, and other build tools we tested (See build performance for details). These changes were contributed by Ethan Atkins (@eatkins).

Glob

sbt 1.3.0 introduces a new type, Glob, that describes a path search query similar to Unix shell globs. For example, all of the scala sources in the project directory can be described by Glob(baseDirectory.value, RecursiveGlob / "*.scala") or baseDirectory.value.toGlob / ** / "*.scala", where ** is an alias for RecursiveGlob. Glob expands on PathFinders but they can be composed with no io overhead. Globs can be retrieved using a FileTreeView. For example, one can write:

val scalaSources = baseDirectory.value.toGlob / ** / "*.scala"
val javaSources = baseDirectory.value.toGlob / ** / "*.java"
val allSources = fileTreeView.value.list(Seq(scalaSources, javaSources))

and the FileTreeView will only traverse the base directory once. Globs and FileTreeView were added by Ethan Atkins (@eatkins) in io#178io#216, and io#226

Watch improvements

sbt 1.3.0 introduces a new file monitoring implementation. It uses enhanced APIs for tracking file change events using os events. It adds a new parser that extracts the specific task(s) for which it will monitor source files and rerun when it detects changes. Only source dependencies of the running tasks are monitored. For example, when running ~compile, changes to test source files will not trigger a new build. Between file events, there are also now options to return to the shell, rerun the previous command(s) or exit sbt. These changes were implemented by Ethan Atkins (@eatkins) in io#178#216#226#4512, and #4627.

Build definition source watch

sbt 1.3.0 automatically watches the build definition sources and displays a warning if you execute a task without reloading. This can be configured to reload automatically as follows:

Global / onChangedBuildSource := ReloadOnSourceChanges

This feature was contributed by Ethan Atkins (@eatkins) in #4664

Custom incremental tasks

sbt 1.3.0 provides support to implement custom incremental tasks based on files. Given a custom task that returns java.nio.file.Path, Seq[java.nio.file.Path], File, or Seq[File], you can define a few helper tasks to make it more incremental.

import java.nio.file._
import scala.sys.process._
val gccCompile = taskKey[Seq[Path]]("compile C code using gcc")
val gccHeaders = taskKey[Seq[Path]]("header files")
val gccInclude = settingKey[Path]("include directory")
val gccLink = taskKey[Path]("link C code using gcc")

gccCompile / sourceDirectory := sourceDirectory.value
gccCompile / fileInputs += (gccCompile / sourceDirectory).value.toGlob / ** / "*.c"
gccInclude := (gccCompile / sourceDirectory).value.toPath / "include"
gccHeaders / fileInputs += gccInclude.value.toGlob / "*.h"
gccCompile / target := baseDirectory.value / "out"

gccCompile := {
  val objectDir = Files.createDirectories((gccCompile / target).value.toPath / "objects")
  def objectFile(path: Path): Path =
    target.value.toPath / path.getFileName.toString.replaceAll(".c$", ".o")
  Files.createDirectories(target.value.toPath)
  val headerChanges = gccHeaders.inputFileChanges.hasChanges
  val changes = gccCompile.inputFileChanges
  changes.deleted.foreach(sf => Files.deleteIfExists(objectFile(sf)))
  val sourceFileChanges = changes.created ++ changes.modified
  val needRecompile = (sourceFileChanges ++ (if (headerChanges) changes.unmodified else Nil)).toSet

  val logger = streams.value.log
  gccCompile.inputFiles.map { sf =>
    val of = objectFile(sf)
    if (!Files.exists(of) || needRecompile(sf)) {
      logger.info(s"Compiling $sf")
      s"gcc -I${gccInclude.value} -c $sf -o $of".!!
    }
    of
  }
}

Given this setup, gccCompile.inputFiles will return a sequence of all of the input c source files, gccCompile.inputFileChanges returns a data structure containing the created, deleted, modified and unmodified files since the last run of gccCompile while gccHeaders.changedInputFiles returns the headers that have changed since the last run of gccCompile. Taken together, these tasks can be used to incrementally only rebuild the source files that need to be rebuilt given the file system changes since the last time gccCompile completed.

In another task such as gccLink, the result of gccCompile can be tracked as well using gccCompile.outputFileChanges.

gccLink := {
  val library = (gccCompile / target).value.toPath / "libmylib.dylib"
  val objectFiles = gccCompile.outputFiles
  val logger = streams.value.log
  if (!Files.exists(library) || gccCompile.outputFileChanges.hasChanges) {
    logger.info(s"Rebuilding $library")
    s"gcc -dynamiclib -o $library ${objectFiles mkString " "}".!!
  }
  library
}

The inputs of a task will automatically be monitored by the command which has a new parser that is context aware. A custom clean task is also implemented for any task that generates file outputs. The clean tasks are aggregated across the project and config scopes. For example, Test / clean will clean all of the files generated by tasks in the Test config declared in the Test config but not the files generated in the Compile config.

This feature was contributed by Ethan Atkins (@eatkins) in #4627.

Super shell

When running in an ANSI-compatible terminal, sbt 1.3.0 will display the currently running tasks. This gives the developer the idea of what tasks are being processed in parallel, and where the build is spending its time. In homage to Gradle's "Rich Console" and Buck's "Super Console", we call ours "Super shell."

To opt-out put the following in the build:

ThisBuild / useSuperShell := false

or run sbt with --supershell=false (or -Dsbt.supershell=false). This feature was added by Eugene Yokota (@eed3si9n) as #4396/util#196.

Tracing

To view the task breakdown visually, run sbt with --traces (or -Dsbt.traces=true). This will generate build.traces file, which is viewable using Chrome Tracing chrome://tracing/. This feature was contributed by Jason Zaugg (@retronym).

To output the task timings on screen, run sbt with --timings (or -Dsbt.task.timings=true -Dsbt.task.timings.on.shutdown=true).

SemanticDB support

sbt 1.3.0 makes it easier to generate SemanticDB. To enable the generation of SemanticDB build-wide:

ThisBuild / semanticdbEnabled := true
ThisBuild / semanticdbVersion := "4.1.9"
ThisBuild / semanticdbIncludeInJar := false

This was added by @eed3si9n as #4410.

print command

sbt 1.3.0 adds a new print command, similar to show but prints directly to standard out.

# sbt -no-colors --error  "print akka-cluster/scalaVersion"
2.12.8

This was contributed by David Knapp (@Falmarri) as #4341

Appending Function1

Function1 can be appened using +=.

Global / onLoad += { s =>
  doSomething()
  s
}

This was contributed by Dale Wijnand (@dwijnand) as #4521.

JDK 11 support

sbt 1.3.0 is first release of sbt that's been testing on JDK11 extensively.
All integration tests on Travis CI are on AdoptOpenJDK's JDK 11, which were updated by @eed3si9n as #4389/zinc#639/zinc640.

  • Fixes warnings on JDK 9+ by upgrading to protobuf 3.7.0 zinc#644 by @smarter
  • Fixes spurious rebuilds caused by invalidation of rt.jar on JDK 11 #4679 by @eatkins

 

Other bug fixes and improvements

  • Fixes cross building with a single-letter alias #4355 / #1074 by @eed3si9n
  • Removes old warning about global directory #4356 / #1054 by @eed3si9n
  • Improves JDK discovery for cross-JDK forking #4313 / #4462 by @raboof
  • Expands ~ in -Dsbt.global.base property to user home. #4367 by @kai-chi
  • Adds def sequential[A](tasks: Seq[Initialize[Task[A]]]): Initialize[Task[A]]. #4369 by @3tty0n
  • Fixes sbt server to send error event on command failure. #4378 by @andreaTP
  • Implements cancellation of request by LSP client. #4384 by @andreaTP
  • Implements "sbt/completion" command in sbt to server to complete sbt commands. #4397 by @andreaTP
  • Fixes errors order reported by sbt server. #4497 by @tdroxler
  • Fixes cached resolution. #4424 by @eed3si9n
  • The sbt task definition linter warns rather than errors by default. The linter can be disabled entirely by putting import sbt.dsl.LinterLevel.Ignore in scope. #4485 by @eatkins
  • Full GC is only automatically triggered when sbt has been idle for at least a minute and is only run at most once between shell commands. This improves shell responsiveness. #4544 by @eatkins
  • Avoids NPE in JDK12. #4549 by @retronym
  • Fixes the eviction warning summary lm#288 by @bigwheel
  • Fixes Zinc's flag to skip the persistence of API info. zinc#399 by @romanowski
  • Fixes Zinc not detecting synthetic top level member changes. #4316/zinc#572 by @jvican
  • Zinc to notify callback of generated non-local classes before the compiler's middle and backend phases. zinc#582 by @jvican
  • Removes a use of regex in Zinc for performance. zinc#583 by @retronym
  • Fixes incremental compilation involving default arguments. zinc#591 by @jvican
  • Adds Analysis callback of Zinc thread-safe. zinc#626 by @dotta
  • Fixes a non-zero exit Javadoc not failing the task. zinc#625 by @raboof
  • Fixes stack trace showing up when tasks fail sbt/util#211
  • Turns off Gigahorse by default since keep-alive causes issue while publishing to Nexus or Artifactory sbt/librarymanagement#317
  • Some functions under sbt.internal.inc.ZincUtil that uses LM have moved to ZincLmUtil to break LM dependency from Zinc. zinc#655 by @dwijnand

Participation

First, I'd like to introduce Ethan Atkins, a core community member of the sbt project, and author of Close Watch, which uses native code to provide watch service on macOS. Normally I don't publicize the number of commits, but here's the top 10 for sbt 1.3.0:


541 Ethan Atkins
369 Eugene Yokota (eed3si9n)
42  Jorge Vicente Cantero (jvican)
35  Łukasz Wawrzyk
34  Dale Wijnand
24  Andrea Peruffo
16​​  Kenji Yoshida (xuwei-k)
13  Guillaume Martres
7   Arnout Engelen
7   Jason Zaugg

As a community member, Ethan has contributed various IO related improvements to make sbt more responsive in his own time. sbt 1.3.0 reflects many of his ideas.

The last feature release of sbt 1 was sbt 1.2.0 in July, 2018. Since then, we've released eight patch releases under sbt 1.2.x for bug fixes, but most of the feature enhancements were merged to develop branch. Over the course of these months, 45 contributors contributors participated in sbt 1.3.0 and Zinc: Ethan Atkins, Eugene Yokota (eed3si9n), Jorge Vicente Cantero (jvican), Łukasz Wawrzyk, Dale Wijnand, Andrea Peruffo, Kenji Yoshida (xuwei-k), Guillaume Martres, Arnout Engelen, Jason Zaugg, Krzysztof Romanowski, Antonio Cunei, Mirco Dotta, OlegYch, Alex Dupre, Nepomuk Seiler, 0lejk4, Alexandre Archambault, Eric Peters, Kazuhiro Sera, Philippus, Som Snytt, Syed Akber Jafri, Thomas Droxler, Veera Venky, bigwheel, Akhtyam Sakaev, Alexey Vakhrenev, Eugene Platonov, Helena Edelson, Ignasi Marimon-Clos, Julien Sirocchi, Justin Kaeser, Kajetan Maliszewski, Leonard Ehrenfried, Mikołaj Jakubowski, Nafer Sanabria, Stefan Wachter, Yasuhiro Tatsuno, Yusuke Izawa, falmarri, ilya, kai-chi, tanishiking, Ólafur Páll Geirsson. Thank you!

Thanks to everyone who's helped improve sbt and Zinc 1 by using RCs and reporting bugs, improving our documentation, porting builds, porting plugins, and submitting and reviewing pull requests. For anyone interested in helping sbt, there are many avenues for you to help, depending on your interest. If you're interested, Contributing, sbt-contrib, "help wanted", "good first issue" are good starting points.

Share



Comments


View All Posts or Filter By Tag


Questions?