Today, we're pleased to announce the open source availability of dbuild, a tool we use within Typesafe to test Scala's ecosystem. dbuild is a tool which can take a disparate set of projects and compile them against each other, ensuring that modifications to code in one project does not break code in another project. dbuild was instrumental in the Scala 2.11 release, where it was used to compile and validate more than 70 open source projects, 1.2 million lines of Scala code and 120 thousand lines of Java code. This helped ensure that our open source ecosystem would have a smooth transition from Scala 2.10 into the 2.11 series.

What is dbuild?

dbuild is not a build tool, it is a build aggregator. dbuild takes a set of distinct projects, like Scala, scalaz, Akka, spark, and attempts to rewire their builds such that all code is built together, as a cohesive unit. For example, we can build the latest in-development Scala, then use this recently built Scala to build and test a version of Akka. Finally, this newly built version of Akka is used to build and test the Play project.

Here's an example dbuild configuration which will build a few of the javascript plugins being actively developed for Play 2.3 and ensure they are compatible

build: {
 projects: [{
     name : sbt-web
     uri : "https://github.com/sbt/sbt-web.git"
  }, {
     name : js-engine
     uri  : "https://github.com/typesafehub/js-engine.git"
  }, {
     name: sbt-less
     uri: "https://github.com/sbt/sbt-less.git"
  }]
}

Here, three projects: sbt-web, js-engine + sbt-less are compiled together to ensure their code and APIs are compatible.

Why dbuild?

The Scala ecosystem is composed of a rich and diverse set of projects. Each project has its own community and policies governing its development. It would be impossible to try to lock a community down into one build tool, and one standard of how to publish/consume artifacts.

At the same time, every Scala major release generally involved a bit of ceremony around ensuring the ecosystem projects were compatible with the latest version of Scala, and any changes/deprecations that may have occurred. It became apparent that we needed better tooling to simplify/ease the transition of the Scala ecosystem from one major release to the next.

dbuild does many things for the Scala ecosystem:

  • It tests nightly Scala changes against a significant portion of the OSS Scala project to avoid regressions
  • It helps the Scala developers reach out to community project owners when deprecations (or removal of deprecated methods) affect them.
  • It helps ensure that when a major version of Scala is released, the libraries our users have come to love are also available/ready for release against the latest version.
  • It lets us experiment with Scala features and have confidence on the impact they have to the community.

Who should use dbuild?

While, at Typesafe, we use dbuild to verify Scala’s standard library and compiler, it is applicable to any project which has an ecosystem of libraries/plugins which depend on it.    

For example:

  • Ensure Javascript plugins work with latest sbt-web release for Play!
  • Ensure sbt-plugins work with the latest sbt major version.
  • Ensure external Akka modules work with the latest version of akka.

The Scala 2.12 roadmap includes switching over to a new compiler backend & optimizer. The new backend has been developed in a fork of Scala compiler created by Miguel Garcia from EPFL. Miguel’s fork replaces about 20% of the entire Scala compiler code base. Merging back such a large changeset and avoiding the risk of introducing regressions poses a significant challenge. We are using dbuild and community builds configuration to build 1.2 million lines of Scala code and verify that changes to the backend are not breaking existing Scala projects. This method of massive building and testing allowed us to uncover tricky bugs that were not caught by Scala’s standard test suite.

Where can I find out more?

dbuild is offered for end users, with install instructions in the user manual. Additionally, OSS contributors interested in hacking on the core should check out the github repository: https://github.com/typesafehub/dbuild

Enjoy!

Share



Comments


View All Posts or Filter By Tag