lots of resolver errors in build logs (maven.twttr.com) about community-build HOT 10 CLOSED

Following up on chat on https://gitter.im/typelevel/general?at=57136f222c9711166432a27e

I'm seeing constant local build time of around 2h 30 min, mainly achieved by using artifactory with virtual repositories, as per #152 (comment)

The setup is really rather easy:

Just add the extra repositories listed in common.conf to artifactory as Maven/Ivy/sbt remote repositories (Make sure "Suppress POM Consistency Checks" is ticked). At this point, you have just proxied some extra repos.

Then create three virtual repositories, such as (CB-Ivy, CB-Maven, CB-Sbt) and add the current repositories listed in common.conf to the appropriate Virtual repo, in the same order. The idea here is that all the drop down through the repos is "pre-calculated"

Your repositories in common.conf is then something like:

 local
  cb-ivy:   YOUR-URL/artifactory/CB-Ivy, [organization]/[module]/(scala_[scalaVersion]/)(sbt_[sbtVersion]/)[revision]/[type]s/[artifact](-[classifier]).[ext]
  cb-maven: YOUR-URL/artifactory/CB-Maven
  cb-sbt:   YOUR-URL/artifactory/CB-Sbt, [organization]/[module]/(scala_[scalaVersion]/)(sbt_[sbtVersion]/)[revision]/[type]s/[artifact](-[classifier]).[ext] 

If needed, you could always create the Virtual repos per config - eg "CB-Ivy_2.12.x" and so on.

I already had Artifactory setup, but not used by default and it did help adding it to dbuild, but the dbuilds were still really slow. ( as in Ctrl-C slow...).

Adding the Virtual repos was the cure :)

from community-build.

Should be resolved by #243 and #244 (via scala/scala-jenkins-infra#171)

from community-build.

Note that this moves all the resolver config into artifactory, whose config isn't under source control... (scala/scala-jenkins-infra#172)

from community-build.

ugh, well, it looks like now the slowness turned into resolver issues

from community-build.

Looking into this, it seems the maven.twttr.com repo is returning jars using the text/html mimetype? E.g., https://maven.twttr.com/com/twitter/common/objectsize/0.0.10/objectsize-0.0.10.jar is showing up as text in my browser, and artifactory refuses to cache it. (https://scala-ci.typesafe.com/artifactory/twitter-maven/com/twitter/common/objectsize/0.0.10/objectsize-0.0.10.jar says:

{
  "errors" : [ {
    "status" : 409,
    "message" : "Error fetching https://maven.twttr.com/com/twitter/common/objectsize/0.0.10/objectsize-0.0.10.jar: Mismatching mime types found in request: [application/java-archive] and response: text/html, and this repo blocks text/html on mismatch (remote response: 409: null)"
  } ]
}

)

from community-build.

@stuhood is this something someone at Twitter could address? serving up artifacts with the wrong MIME type seems like something that could affect a variety of downstream tools, not just the Scala community build in particular

from community-build.

the problem on the Twitter end still exists afaik (I didn't check again just now), but we don't care so much at our end now since #243, so I'm closing this

from community-build.

fyi @mosesn @kevinoliver

from community-build.

@SethTisue thanks, we're aware of the problem but don't have an excellent solution yet.

from community-build.

recently: a team at Twitter "took it upon themselves to migrate their dependencies to maven central. The need to use maven.twttr.com should go away entirely soon"

Adriaan: "We'll drop the maven.twttr.com repo from our artifactory cache once the move is complete"

new ticket: scala/scala-jenkins-infra#197

from community-build.