Link conda packages to libhdf5 about conda-recipes HOT 4 CLOSED

Thanks for writing this up @speth. conda-forge takes the approach that everything is built against the same version of these dependencies, so upgrades aren't as much of a problem. I think this is closest to your version 1 above.

Option 2 is feasible because conda build automatically includes the dependency in the hash part of the version string, so setting that wouldn't have to be done manually. The trick would be compiling against multiple libhdf5 versions, which will take forever.

The third option is probably the one with the least maintenance burden, unless ...

What if we only distribute conda packages via conda-forge? It would certainly save a lot of effort maintaining two recipes and build systems. It would (maybe) eliminate some confusion about differences between the channels (not that I've seen any on the UG). It would also eliminate this class of problem, as long as the recipe was kept up-to-date as new dependency libraries are released. This is generally handled by automated pull requests. For example, Cantera would have to support the newest, say, sundials when it came out I think. However, there'd be more support from CF team members who know conda and compilers and their crossover better than any of us.

The other fly is that we don't have a Windows package on CF yet. It could be added, but I'm not sure how hard that would be. I don't recall why we gave up on that however long ago we started with CF.

What do you think?

from conda-recipes.

Thank you both for the insights. Based on @bryanwweber’s arguments, limiting everything to CF may be the easier option long term. The other issue is that Intel mac’s are being replaced by Apple Silicon, which also supports a move towards CF.

from conda-recipes.

Thanks for elaborating on this, @bryanwweber.

conda-forge takes the approach that everything is built against the same version of these dependencies, so upgrades aren't as much of a problem. I think this is closest to your version 1 above.

So what happens when a new version of the dependency is released? They trigger new builds of all dependent packages?

The trick would be compiling against multiple libhdf5 versions, which will take forever.

In principle, this could be avoided by letting SCons do incremental builds. For a different HDF5 version, only a single source file needs to be recompiled and then linked into a different version of the Cantera library file. I don't know if this is possible within conda-build, though. I guess one mitigating factor is that now that we're dynamically linking the Python module to the Cantera shared library, only the shared library depends on HDF5 directly. So the number of distinct builds/packages is just the sum of the HDF5 and Python versions, not the product.

What if we only distribute conda packages via conda-forge? It would certainly save a lot of effort maintaining two recipes and build systems. It would (maybe) eliminate some confusion about differences between the channels (not that I've seen any on the UG).

I have a couple of potential concerns here.

First, I think it's very useful to be able to test the conda packaging infrastructure and resulting packages for development versions. As I think you've explained, conda-forge only builds packages for release versions of a package, which means that we don't get to test the builds until after we've made a Cantera release, and any fixes that need to be made in Cantera either need to be handled as build-time patches (always tedious) or as a subsequent minor release of Cantera.

I also think it's useful for us to be able to distribute alpha/beta conda packages for testing purposes, and for users that like to stay closer to the cutting edge without always having to compile Cantera themselves.

Last, I thought that packages from the defaults channel and conda-forge weren't supposed to be mixed, or at least were likely to lead to lots of version conflicts. So if we don't build for the defaults channel, then we won't have a Cantera package available for users of mainline Anaconda (unless the PyPI package suffice here?).

Perhaps the first two of these could be mitigated by running CI ourselves on the recipe, and uploading those packages to a different conda channel?

It would also eliminate this class of problem, as long as the recipe was kept up-to-date as new dependency libraries are released. This is generally handled by automated pull requests. For example, Cantera would have to support the newest, say, sundials when it came out I think.

At least in the case of Sundials, that will often require a new Cantera release. I guess I'm not sure what the state of this is right now for the conda-forge packages and Cantera 2.6.

To Ingmar's comment:

The other issue is that Intel mac’s are being replaced by Apple Silicon, which also supports a move towards CF.

GH Actions runners for macOS arm64 are on the way, expected by the end of this year (see here, though I'm not sure I want to build for "Apply silicone" 🤣 ). For this release, I'm wondering if it's worth just building those packages locally and uploading them, given that my development machine is an M1 anyway.

from conda-recipes.

So what happens when a new version of the dependency is released? They trigger new builds of all dependent packages?

Yes, see conda-forge/cantera-feedstock#26 as one example

At least in the case of Sundials, that will often require a new Cantera release. I guess I'm not sure what the state of this is right now for the conda-forge packages and Cantera 2.6.

At least for now, it's pinned to sundials ==5.7: https://github.com/conda-forge/cantera-feedstock/blob/main/recipe/meta.yaml#L39 I guess that might limit versions of other dependencies, but I'm not sure what other packages depend on sundials to do a reverse dependency check.

conda-forge only builds packages for release versions of a package,

It should be possible to build versions while we're testing updates for an upcoming release. I think the policy is that they only want released versions uploaded to the Anaconda servers for distribution. For instance, merging to a feature branch prior to an upcoming release should be OK, I would think.

I also think it's useful for us to be able to distribute alpha/beta conda packages for testing purposes,

Yeah, we would lose this ability I suppose. We'd have to balance this against the maintenance needs of having our own recipe.

So if we don't build for the defaults channel, then we won't have a Cantera package available for users of mainline Anaconda (unless the PyPI package suffice here?).

This is also true, and the PyPI package would have the same problem potential set of problems that the conda forge package would have regarding compiler versions and ABI compatibility. There are about 140k downloads of 2.6.0 from conda forge and 100k from mainline Anaconda source. OTOH, there are 58.5M downloads of CF numpy and only 4.5M of mainline Anaconda source. I'm not sure that's helpful...

Perhaps the first two of these could be mitigated by running CI ourselves on the recipe, and uploading those packages to a different conda channel?

I think this could be possible, although one of the big benefits of CF is their automated setup of the infrastructure required for all these builds, which we'd have to somehow duplicate for at least a subset of the build matrix.

So the number of distinct builds/packages is just the sum of the HDF5 and Python versions, not the product.

This is encouraging! 🎉

In the end, to me, it's a balance of maintenance cost vs. benefit. In my opinion, there'd be a bigger benefit from having you and Ingmar focus more on developing Cantera and less on packaging it. YMMV 😄 This would also possibly be a chance to post a message out on the UG asking for help maintaining our recipe. If it's important to people, they can step up.

from conda-recipes.