Pkg3: Separated Test Environments about juleps HOT 6 OPEN

There should probably be a test-only manifest file, or a test-only section of the main manifest file. test/REQUIRE should be made more capable the same way as REQUIRE

from juleps.

I think this is better handled by staged tags. Nothing should depend on "master," that's a moving target and bad for reproducibility. If you depend on an unreleased sha, then you're not testing against what your users will be installing once it gets released. So this is inherently a temporary thing, and would be better handled by directing the test version of the package to use a test registry to get its dependency version info. Then if the changes from the test registry make it through as is to the release registry, you're testing the same thing users will get.

from juleps.

Nothing should depend on "master," that's a moving target and bad for reproducibility.

Yes, for tagged versions I agree. But that's irrelevant. But I was wondering if it can be standardized how/where branches are chosen for testing. When checking someone's master or dev branch, this stuff can be hidden: it's some script called in the .yml, or sprinkled into the master for the tests. It's up to the developer to pull this all out / comment it out when changes in downstream packages occur and when you eventually want to tag. But it also means that this information is scattered around the repositories if you're trying to find out how to get some development branch working. If it was just in the test/REQUIRE file how everything is being tested, it would be clear (and it would be clear that everything needs to be testing on releases for any tag).

from juleps.

There could be test/{Config,Manifest}.toml files corresponding to the current test/REQUIRE.

from juleps.

if we're using a richer format here, then no need for it to be a separate file really. there would be some advantage to recording test dependency info in the registry for a package as well, so all that information is available without having to download every version of every package to get it

from juleps.

True, it can simply be a separate section. There's space for various kinds of targets but I'm unclear on the design there. @wildart had some thoughts on it so perhaps he can chime in here.

from juleps.