| layout | developer-doc | ||
|---|---|---|---|
| title | Build Tools | ||
| category | infrastructure | ||
| tags |
|
||
| order | 1 |
The project is built using the Scala Build Tool which manages dependencies
between the projects as well as external dependencies and allows for incremental
compilation. The build configuration is defined in
build.sbt.
To help wit build times, we do not want to rebuild the whole project with every change, but to only recompile the files that have been affected by the change. This is handled by sbt which under the hood uses zinc (the incremental compiler for Scala). zinc analyses the compiled files and detects dependencies between them to determine which files have to be recompiled when something has been changed.
As described in Java 11 Migration to
successfully compile the runtime project, the JVM running sbt must use the
overridden JAR for Truffle API. This JAR has to be present during startup of the
JVM, but it has to be downloaded from the Maven repository.
To fix this chicken-and-egg problem, we have a special bootstrap task, that
has to be ran when setting-up the project (and after a version change of Graal).
It makes sure the JAR is downloaded and copied to our directory and terminates
the sbt process to ensure that the user restarts it. Without the full restart,
the JAR would not be seen by the JVM. So when setting up the project or after
changing the version of Graal, before launching the sbt shell, you should first
run sbt bootstrap, to make sure the environment is properly prepared.
The logic for copying the JAR is implemented in the bootstrapJARs task in
CopyTruffleJAR.
There are some invariants that are specific to our project, so they are not tracked by sbt, but we want to ensure that they hold to avoid cryptic errors at compilation or runtime.
To check some state before compilation, we add our tasks as dependencies of
Compile / compile / compileInputs by adding the following to the settings of a
particular project.
Compile / compile / compileInputs := (Compile / compile / compileInputs)
.dependsOn(preCompileHookTask)
.value
Tasks that should be run before compilation, should be attached to the
compileInputs task. That is because the actual compilation process is ran in
the task compileIncremental. Compile / compile depends on
compileIncremental but if we add our dependency to Compile / compile, it is
considered as independent with compileIncremental, so sbt may schedule it to
run in parallel with the actual compilation process. To guarantee that our
pre-flight checks complete before the actual compilation, we add them as a
dependency of compileInputs which runs strictly before actual compilation.
To check some invariants after compilation, we can replace the original
Compile / compile task with a custom one which does its post-compile checks
and returns the result of (Compile / compile).value.
There are additional tasks defined in the project directory.
They are used by build.sbt to provide some additional
functionality.
EnvironmentCheck defines a helper
function that can be attached to the default Global / onLoad state transition
to run a version check when loading the sbt project. This helper function
compares the version of JVM running sbt with GraalVM version defined in
build.sbt and the version of flatc installed in the
system with the Flatbuffers library version defined in
build.sbt. If the versions do not match it reports an error
telling the user to change to the correct version.
BenchTasks defines configuration keys for
benchmarking.
BenchTasks records version information
including what git commit has been used for compiling the project. This
information is used by enso --version.
Truffle annotation processor generates a file that registers instruments provided by the runtime. Unfortunately, with incremental compilation, only the changed instruments are recompiled and the annotation processor does not detect this, so un-changed instruments get overwritten.
In the past we had a pre-compile task (see FixInstrumentsGeneration) that detected changes to instruments and if only one of them was to be recompiled, it forced recompilation of all of them, to ensure consistency. This workaround helped to avoid later runtime issues but sometimes triggered a cascade of recompilations, which weren't clear to the end user. Instead, to avoid overwriting entries in META-INF files, individual services were moved to separate subprojects and during assembly of uber jar we concatenate meta files with the same service name.
GenerateFlatbuffers defines the
task that runs the Flatbuffer compiler flatc whenever the flatbuffer
definitions have been changed. It also makes sure that flatc is available on
PATH and that its version matches the version of the library. It reports any
errors.
As described in Bootstrapping, to successfully compile the
runtime subproject, the JVM running sbt must load some JARs at startup. The
user should run sbt bootstrap to ensure that.
If the compilation proceeds without the bootstrapped JARs it may lead to
inconsistent state with some dependencies being undetected and weird errors. To
avoid such situations, CopyTruffleJAR
defines is a pre-compile task that is executed before compiling the runtime
subproject which makes sure that the Truffle JAR is up-to-date. Even if the
developer forgets about bootstrap, this pre-compile task will update the
Truffle JARs. However, as the JARs are loaded at startup, the JVM has to be
restarted. To force the user to restart it, the current sbt process is
terminated.
This pre-compile task runs before compiling runtime sources, but some
dependencies may have started compiling in the meantime (as sbt schedules
independent tasks in parallel). The fact that these dependencies are up-to-date
may not be registered by sbt due to the abrupt termination. This does not affect
consistency of the build state though - as a result these dependencies may be
recompiled again when the compilation is restarted. This is acceptable as this
restart is just an additional check to ensure correctness and improve user
experience. In normal operation, the restart should never be triggered, as the
user should remember to run bootstrap when necessary.
WithDebugCommand defines a command
that allows to run a task with additional JVM-level flags.
RecompileParser defines a task that can
be attached to the compile task in configurations of the syntax project.
This task ensures that the syntax project is recompiled whenever
syntax-definition changes.
NativeImage task is described at
Native Image.