radically simplify the build about community-build HOT 6 CLOSED

Summary of discussion at team meeting:

General sentiment is favorable.

There was a bit of hesistancy/uncertainty around losing coverage:

• The old setup heavily tests "new compiler generates binary artifact, new compiler then consumes those artifacts". The new setup will instead heavily test "new compiler consumes artifacts generated by previous compilers". Six of one, half a dozen of another?
• None of us could think of a specific bug the old setup caught that would now be missed, but it's hard to be sure.

Conclusion: We might consider keeping a drastically scaled-down, much-easier-to-maintain version of the old setup. Say, ~20 projects instead of ~200? Not clear if it's really necessary.

Jason wondered if the main body of the build ought to be two-pass: build and publish everything using existing binary dependencies, then re-build everything using the freshly built dependencies. We batted this idea around a bit and we didn't rule it out, but it isn't clearly worth doing, either.

There was some question of whether we can get away without rewiring downstream repos' dependencies on compiler plugins. Suppose everyone is depending on different versions of kind-projector? My sense is that there is no fundamental difficulty and that we'll be able to deal with this as follows:

• If a downstream repo is found to be using an out-of-date compiler plugin, we can PR the upgrade (and encourage the maintainers to use Scala Steward for this if they aren't already).
• If we need to fork a downstream repo from time to time we can... lord knows the old setup has often required us to do that 🙀

Dale wondered if it was weird to republish compiler plugins ourselves without changing the coordinates; could that poison artifact caches, including our own? I think we convinced ourselves that it's okay because if a compiler plugin is published with full cross-versioning actually the coordinates won't be identical, because the full Scala version is included in the artifact name, so e.g. if we published kind-projector the artifact name would be e.g kind-projector_2.13.7-bin-abcd123. (And anyway we wouldn't be publishing to Maven Central, only to the scala-ci artifactory.)

from community-build.

proof-of-concept branch: https://github.com/SethTisue/community-build/tree/issue-1456
green proof-of-concept run: https://scala-ci.typesafe.com/job/scala-2.13.x-jdk8-integrate-community-build/4381/

so far so good

from community-build.

Note that with this change, we wouldn't really need dbuild anymore. The GitHub Actions job that publishes Scala nightlies could also publish scalameta and friends to the same Artifactory the nightlies are published to (https://scala-ci.typesafe.com/artifactory/scala-integration/) . Then the main body of the community build could stay on dbuild, or we could replace it with some Scala code (or, god forbid, a shell script) that builds each of the remaining repos separately.

from community-build.

In the longer term, converging with whatever Scala 3 ends up doing in this area remains a possibility. For now, I think this approach is the shortest/easiest path to getting us to a build that 1) meets our current needs, but 2) is easier to maintain.

from community-build.

PR: #1472

from community-build.

done. for now, both layers of the build remain dbuild-based, because it works and it's the path of least resistance since it allowed for the most reuse of what we already had.

Note that if we do this, we could also consider publishing kind-projector, scalameta, genjavadoc, et al to a publicly available resolver. Then almost anyone who wants to test Scala 2 nightly builds in their own CI would be able to do so, whereas today, often they can't

I'm setting this idea aside for now. We can consider, at our leisure, whether we should make it a priority at some point.

from community-build.