Pre-release versions of coiled-runtime about benchmarks HOT 7 CLOSED

n.b.: We'd probably want to having staging.coiledhq.com use the pre-release in its default software environment, which will probably require changes to https://github.com/coiled/software-environments/.

from benchmarks.

It would be very helpful for us to have pre-releases of coiled-runtime available, either cut directly from dask main or a hand selected more recent version.

Sounds good. I figured this would be something we'd want to do at some point.

We could use this internally at coiled on our staging deployment as well to test things before promoting it to a full release

Just curious, how much activity does the staging deployment get? For example, I know there's an integration test suite -- does that run on staging?

from benchmarks.

The integ tests run on both staging and prod.

It's a lot of launching clusters with various options. No serious computations. (Closest thing is we run the taxi quickstart example.)

Currently they fail a lot. The most common reason is that we are using coiled-runtime as the software env, which doesn't include this: dask/distributed#5982.

from benchmarks.

Thanks for the additional info @dchudz

I'll suggest we add a new GHA workflow which runs on a cron and publishes a nightly version of coiled-runtime that uses the nightly of dask and distributed conda packages that are published on the dask conda channel https://anaconda.org/dask. I suspect we can also reuse some of the logic around how those dask and distributed nightlies are generated here for publishing a nightly coiled-runtime.

from benchmarks.

Sounds great, I think. After that someone could shift our GHA's in https://github.com/coiled/software-environments/ to use the nightly for staging.

In the meantime I think @shughes-uk is going to hack the default environment on staging to use latest.

from benchmarks.

In the meantime I think @shughes-uk is going to hack the default environment on staging to use latest.

Just checking in -- was a temporary hack possible? I'm thinking about prioritization and wanted to get a sense for how critical getting a pre-release process out is

from benchmarks.

@jrbourbeau yes I hacked it in. My highest priority ask now is just getting a new version of dask/distributed out and into coiled runtime, the stability difference is huge with the very minor plugin tweak!

from benchmarks.