Add JDK 21 to build matrix about community-build HOT 22 CLOSED

gah. this is too much hassle. let's take this up after 2.12.18 is out and we have an sbt release that uses it.

JDK 21 is still EA, there's still plenty of time to fix any further issues, if any crop up.

from community-build.

https://github.com/adoptium/temurin21-binaries/ doesn't have any binaries yet, but it should be fine to use the ones at https://jdk.java.net/21/

from community-build.

• behemoth 1
• behemoth 2
• behemoth 3

steps

• cd /usr/lib/jvm
• sudo curl -O -L 'https://download.java.net/java/early_access/jdk21/22/GPL/openjdk-21-ea+22_linux-x64_bin.tar.gz'
• sudo dtrx -n openjdk-21-ea+22_linux-x64_bin.tar.gz
• sudo rm openjdk-21-ea+22_linux-x64_bin.tar.gz
• sudo chmod -R go+rX openjdk-21-ea+22_linux-x64_bin
• sudo ln -s openjdk-21-ea+22_linux-x64_bin/jdk-21 java-21-openjdk-x64
• ./java-21-openjdk-x64/bin/java -version

openjdk version "21-ea" 2023-09-19
OpenJDK Runtime Environment (build 21-ea+22-1890)
OpenJDK 64-Bit Server VM (build 21-ea+22-1890, mixed mode, sharing)

from community-build.

made configs:

• https://scala-ci.typesafe.com/job/scala-2.12.x-jdk21-integrate-community-build/
• https://scala-ci.typesafe.com/job/scala-2.13.x-jdk21-integrate-community-build/

from community-build.

I triggered test runs, but they're (as expected) failing because sbt is using Scala 2.12.17 which doesn't work on JDK 21.

Can I find a way to have dbuild start sbt with sbt -Dsbt.scala.version=2.12.18-M2? If so, I can move forward with this immediately.

If not, we'll have to:

• wait until 2.12.18 is out and Eugene releases an sbt version that defaults to it
• accept that any projects that require older sbt versions will be expected-to-fail

from community-build.

Can I find a way to have dbuild start sbt with sbt -Dsbt.scala.version=2.12.18-M2?

looks like this will work:

community.conf
-  sbt-java-options: ["-Xms1536m", "-Xmx1536m", "-Xss2m"]
+  sbt-java-options: ["-Xms1536m", "-Xmx1536m", "-Xss2m", "-Dsbt.scala.version=2.12.18-M2"]

from community-build.

some repos (e.g. specs2) will hit this: [error] sbt.librarymanagement.ResolveException: Error downloading org.scoverage:scalac-scoverage-plugin_2.12.18-M2:1.4.8

probably enough repos will avoid this that we'll get some reasonably satisfactory level of testing now? with more to come later once we're off 2.12.8-M2 and onto 2.12.18. it's fine, 21 is still early-access

from community-build.

some repos (e.g. specs2) will hit this: [error] sbt.librarymanagement.ResolveException: Error downloading org.scoverage:scalac-scoverage-plugin_2.12.18-M2:1.4.8

and that means we can't use -Dsbt.scala.version=2.12.18-M2 on all JDKs. we'll need to make it configurable via Jenkins so we can use it on the JDK 21 runs only

from community-build.

and the scalac-scoverage-plugin problem prevents projects from even extracting :-(

so this isn't actually going to be mergeable until after 2.12.18 is out. however, I could still do some runs on branches, using a restricted set of projects

from community-build.

2.12.18 and sbt 1.9.0 are out

on 2.13.x, it looks like we only need a few small changes: #1671

2.12.x will be harder (too hard, perhaps) since there are still sbt 0.13.x projects involved

from community-build.

2.13.x is green after one more change (63a883b) — same projects fail as on JDK 20

for 2.12.x, I guess I'll make a branch with a subset

from community-build.

for 2.12.x, I guess I'll make a branch with a subset

that subset could end up being quite small, since the list of sbt 0.13 based projects:

akka-contrib-extra
autowire
case-app
elastic4s
export-hook
fastparse
jackson-module-scala
jawn-0-10
lift-json
linter
macro-compat
pcplod
sbt-testng
scala-collection-laws
scala-debugger
scaladex
scalameta
scalameter
scalamock
scalasti
scalastyle
scalatest
sksamuel-exts
upickle

includes projects such as scalatest (158 downstream projects) and macro-compat (61 downstream projects)

some possible ways forward:

• accept that the subset will be quite small (it would still be enough to verify basic functioning of the compiler, standard library, and sbt 1)
• get scalatest and macro-compat building somehow on sbt 1 (macro-compat probably not hard; scalatest probably a mess I'd prefer not to tackle, but I could at least try)
• use a binary dependency for any project we can't build

probably using binary dependencies as needed makes the most sense?

from community-build.

I don't think it will be possible for me to tackle this until August

from community-build.

Since we have the 2.13.12 release coming up, today I upgraded all 3 behemoths to JDK 21 EA build 35 (dated August 11, from https://jdk.java.net/21/):

• behemoth 1
• behemoth 2
• behemoth 3

steps

cd /usr/lib/jvm
sudo curl -O -L 'https://download.java.net/java/GA/jdk21/fd2272bbf8e04c3dbaee13770090416c/35/GPL/openjdk-21_linux-x64_bin.tar.gz'
sudo dtrx -n openjdk-21_linux-x64_bin.tar.gz
sudo mv openjdk-21_linux-x64_bin openjdk-21-ea+35_linux-x64_bin
sudo rm openjdk-21_linux-x64_bin.tar.gz
sudo chmod -R go+rX openjdk-21-ea+35_linux-x64_bin
sudo rm java-21-openjdk-x64
sudo ln -s openjdk-21-ea+35_linux-x64_bin/jdk-21 java-21-openjdk-x64
./java-21-openjdk-x64/bin/java -version

openjdk version "21" 2023-09-19
OpenJDK Runtime Environment (build 21+35-2513)
OpenJDK 64-Bit Server VM (build 21+35-2513, mixed mode, sharing)

from community-build.

I'd like to get back on all-Temurin, just for consistency's sake, so I'll upgrade the behemoths once 21 appears at https://adoptium.net/temurin/releases/

from community-build.

this seems like confirmation that waiting for official Temurin is okay (and won't take too long): https://fosstodon.org/@sxa/111098188725062282

from community-build.

21 is here! https://adoptium.net/temurin/releases/

from community-build.

behemoths

• behemoth 1
• behemoth 2
• behemoth 3

steps

• ssh admin@jenkins-worker-behemoth-<n>
• cd /usr/lib/jvm
• sudo rm java-20-openjdk-x64
• sudo rm -rf OpenJDK20U-jdk_x64_linux_hotspot_20_36
• sudo rm -rf openjdk-21-ea+22_linux-x64_bin
• sudo rm -rf openjdk-21-ea+35_linux-x64_bin
• sudo rm java-21-openjdk-x64
• sudo curl -O -L 'https://github.com/adoptium/temurin21-binaries/releases/download/jdk-21%2B35/OpenJDK21U-jdk_x64_linux_hotspot_21_35.tar.gz'
• sudo dtrx -n OpenJDK21U-jdk_x64_linux_hotspot_21_35.tar.gz
• sudo rm OpenJDK21U-jdk_x64_linux_hotspot_21_35.tar.gz
• sudo chmod -R go+rX OpenJDK21U-jdk_x64_linux_hotspot_21_35
• sudo ln -s OpenJDK21U-jdk_x64_linux_hotspot_21_35/jdk-21+35 java-21-openjdk-x64
• ./java-21-openjdk-x64/bin/java -version

result

openjdk version "21" 2023-09-19 LTS
OpenJDK Runtime Environment Temurin-21+35 (build 21+35-LTS)
OpenJDK 64-Bit Server VM Temurin-21+35 (build 21+35-LTS, mixed mode, sharing)

trigger builds

• https://scala-ci.typesafe.com/view/scala-2.12.x/job/scala-2.12.x-jdk21-integrate-community-build/
• https://scala-ci.typesafe.com/view/scala-2.13.x/job/scala-2.13.x-jdk21-integrate-community-build/

the console log should show the new version near the top

delete old JDK 20 jobs

• https://scala-ci.typesafe.com/view/scala-2.12.x/job/scala-2.12.x-jdk20-integrate-community-build/
• https://scala-ci.typesafe.com/view/scala-2.13.x/job/scala-2.13.x-jdk20-integrate-community-build/

update wiki

https://github.com/scala/community-build/wiki#what-variations-of-the-build-exist

from community-build.

upgrade to Temurin 21 final is complete

leaving this open since I still need to figure out what to do, if anything, about the 2.12.x build

from community-build.

Since I've been working on this at widely spaced intervals, I'd forgotten what the current set of blockers might be.

Some of the above remarks were made before 2.12.18 was out. But it is out now, and scalac-scoverage-plugin is available for it. So perhaps it's possible to at least make everything extract if we -Dsbt.scala.version=2.12.18. At https://scala-ci.typesafe.com/view/scala-2.12.x/job/scala-2.12.x-jdk21-integrate-community-build/195/console , we see:

EXTRACTION FAILED (failed: breeze, catalysts, coursier, curryhoward, doodle, droste, enumeratum, fs2, geny, gigahorse, http4s, mdoc, monix, paradox, play-doc, play-webgoat, playframework, sbt, sbt-util, scala-ssh, specs2, specs2-more, tsec, zinc)

so the sbt 0.13 based projects are extracting; the ones that fail to extract are the ones stuck on sbt 1.3, 1.4, or 1.5.

Hmm, but the only versions of scalac-scoverage-plugin published for 2.12.18 are 2.0.10 and 2.0.11, which are quite now, so of course these older projects aren't using that new of a version number. So I'd have to fork each project and either upgrade or remove scoverage. (And I wonder if upgrading would even work? Perhaps the latest scalac-scoverage-plugin uses sbt APIs that are too new? Maybe it doesn't matter, since it's simplest to just remove the plugin.)

As a test, let me see if I can get specs2 extracting by removing scoverage from our fork (many of these projects are probably already forked) without breaking it on earlier JDKs.

Yay, that worked.

Next: add -Dsbt.scala.version=2.12.18 to the non-extracting projects, then see how many fail due to scoverage (or other compiler plugins such as macro paradise).

from community-build.

Hmm, that prevents some projects from extracting even on lower JDKs: EXTRACTION FAILED (failed: coursier, curryhoward, enumeratum, sbt, sbt-util)

coursier needed a macro paradise version bump (but that alone wasn't enough)

curryhoward needed scoverage removed

that leaves the following projects with a weird java.lang.AbstractMethodError: coursier, enumeratum, sbt, sbt-util

enumeratum and coursier I was able to get onto sbt 1.9.7 in forks

sbt-util and sbt, I'll try the same

Yay, everything extracts now (on 21, and on <21)! We'll see if there are any build failures.

from community-build.

Closing as complete, since I can deal with any remaining build failures individually (by fixing them, or by marking them as expected-to-fail) — it's known that the fundamental issues are solved.

from community-build.