When compile times become a problem, how can Scala developers reason about the relation between their code and compile times?

The goal of this proposal is to allow Scala developers to optimize their codebase to reduce compile times, spotting inefficient implicit searches, expanded macro code, and other reasons that slow down compile times and decrease developer productivity.

This repository holds the compiler plugin and a fork of mainstream scalac that will be eventually be merged upstream. This work is prompted by Morgan Stanley's proposal and was approved in our last advisory board.

The required changes to the compiler, Scalac, are the following:

1. Collect all statistics and optimize checks.
2. Initialize statistics per global.
3. Add extra timers and counters.

The project uses a forked scalac version that is used to compile both the compiler plugin and several OSS projects from the community. The integration tests are for now Circe and Monocle, and they help us look into big profiling numbers and detect hot spots and misbehaviours.

If you think a particular codebase is a good candidate to become an integration test, please open an issue.

The proposal is divided into three main areas:

1. Data generation and capture.
2. Data visualisation and comparison.
3. Reproducibility.

How to tackle each of these problems to make the implementation successful?

The generation of data comes from the guts of the compiler. To optimize for impact, the collection of information is done in two different places (a compiler plugin and a forked scalac).

1. A forked scalac with patches to collect profiling information. All changes are expected to be ported upstream.
2. A compiler plugin to get information from the macro infrastructure independently of the used Scalac version.
3. Profiledb readers and writers to allow IDEs and editors to read and write profiledb's.
4. A proof-of-concept vscode integration that displays the data collected from the profiledb.
5. An sbt plugin for reproducibility that warms up the compiler before profiling.

The work is split into two parts so that Scala developers that are stuck in previous Scala versions can use the compiler plugin to get some profiling information about macros.

This structure is more practical because it allow us to evolve things faster in the compiler plugin, or put there things that cannot be merged upstream.

The profiling data will be accessible in two different ways (provided that the pertinent profiling flags are enabled):

1. A summary of the stats will be printed out in every compile run.
2. A protobuf file will be generated at the root of the class files directory.
   • The file is generated via protobuf so that it's backwards and forwards binary compatible
   • The protobuf file will contain all the profiling information.

Why a protobuf file instead of a JSON file? Forwards and backwards binary compatibility is important -- we want our tooling to be able to read files generated by previous or upcoming versions of the compiler. Our goal is to create a single tool that all IDEs and third-party tools use to parse and interpret the statistics from JARs and compile runs.

We're collaborating with Intellij to provide some of the statistics within the IDE (e.g. macro invocations or implicit searches per line). We have some ideas to show this information as heat map in the future.

Getting reproducible numbers is important to reason about the code and identifying when a commit increases or decreases compile times with certainty.

To do so, several conditions must be met: the compiler must be warmed up, the load in the running computer must be low, and the hardware must be tweaked to disable options that make executions non reproducible (like Turbo Boost).

However, this warming up cannot be done in an isolated scenario as Scalac's benchmarking infrastructure does because it doesn't measure the overhead of the build tool calling the compiler, which can be significant (e.g. in sbt).

As a result, reproducibility must be achieved in the build tool itself. The goal of this project is to provide an sbt plugin that warms up the compiler by a configurable amount of time. It also bundles recommendations and tips on how and where to run compilation.

In the following sections, I elaborate on the collected data that we want to extract from the compiler as well as technical details for every section in the original proposal.

Per call-site, file and total:

• How many macros are expanded?
• How long do they take to run?
• How many tree nodes do macros create?

• What's the ratio of generated code/user-defined code?
• How many of these tree nodes are discarded? (is this possible?)

Getting hold of this information requires changes in mainstream scalac.

Per call-site, file and total:

• How many implicit searches are triggered per position?
• How many implicit searches are triggered for a given type?
• How long implicit searches take to run?
• How many implicit search failures are?
• How many implicit search hits are?
• What's the ratio of search failures/hits?

These are the requirements that the proposal lays out.

Note that in some cases, this plugin provides more information than the requested by the original proposal.

• Compilation time totally (this is provided by -Ystatistics)
• Compilation time per file (this is provided by -Ystatistics)
  • Total
  • Broken down by phase
    Note: As of now, the collection of this data has been delayed because it is unknown how it can be properly profiled. Because of pecularities of the typechecker, that jumps from compilation units to compilation units depending on symbol initialization, collecting this data is difficult and it still has to be assessed whether it's possible.
• Times per macro (this is provided by the macro plugin)
  • Per file
  • Per macro
    • Invocations
    • Total time
• Implicit search details (time and number)
  • By type
  • By invocation (only number for now)
  • By file (can be aggregated from the "by invocation" data)
• User time, kernel time, wall clock, I/O time.
  This feature was already provided by Scalac, implemented in this PR.
• Time for flagged features (for certain features – e.g. optimisation)
  • The best way to capture this information is running statistics for the compiler with and without optimization, and compare the profiles. There are also some extra counters.
• Time resolving types from classpath
  • Total
  • By jar: this has been deemed not possible as of the current scalac infrastructure.
• Imports – unused/wildcard timings?
  Assessing the impact of unused and wildcard imports boils down to measuring the cost of creating contexts and scopes in typer and the cost of completing unused symbols. Some informal benchmarks have shown that this cost is actually small, and no compile time difference has been perceived in projects that feature significant amount of unused and wildcard imports. However, in order to enable large-scale examples, our PR to scalac adds counters to Scalac scopes.