Comments (4)
Thanks for writing this up @speth. conda-forge takes the approach that everything is built against the same version of these dependencies, so upgrades aren't as much of a problem. I think this is closest to your version 1 above.
Option 2 is feasible because conda build automatically includes the dependency in the hash part of the version string, so setting that wouldn't have to be done manually. The trick would be compiling against multiple libhdf5 versions, which will take forever.
The third option is probably the one with the least maintenance burden, unless ...
What if we only distribute conda packages via conda-forge? It would certainly save a lot of effort maintaining two recipes and build systems. It would (maybe) eliminate some confusion about differences between the channels (not that I've seen any on the UG). It would also eliminate this class of problem, as long as the recipe was kept up-to-date as new dependency libraries are released. This is generally handled by automated pull requests. For example, Cantera would have to support the newest, say, sundials when it came out I think. However, there'd be more support from CF team members who know conda and compilers and their crossover better than any of us.
The other fly is that we don't have a Windows package on CF yet. It could be added, but I'm not sure how hard that would be. I don't recall why we gave up on that however long ago we started with CF.
What do you think?
from conda-recipes.
Thank you both for the insights. Based on @bryanwweber’s arguments, limiting everything to CF may be the easier option long term. The other issue is that Intel mac’s are being replaced by Apple Silicon, which also supports a move towards CF.
from conda-recipes.
Thanks for elaborating on this, @bryanwweber.
conda-forge takes the approach that everything is built against the same version of these dependencies, so upgrades aren't as much of a problem. I think this is closest to your version 1 above.
So what happens when a new version of the dependency is released? They trigger new builds of all dependent packages?
The trick would be compiling against multiple libhdf5 versions, which will take forever.
In principle, this could be avoided by letting SCons do incremental builds. For a different HDF5 version, only a single source file needs to be recompiled and then linked into a different version of the Cantera library file. I don't know if this is possible within conda-build, though. I guess one mitigating factor is that now that we're dynamically linking the Python module to the Cantera shared library, only the shared library depends on HDF5 directly. So the number of distinct builds/packages is just the sum of the HDF5 and Python versions, not the product.
What if we only distribute conda packages via conda-forge? It would certainly save a lot of effort maintaining two recipes and build systems. It would (maybe) eliminate some confusion about differences between the channels (not that I've seen any on the UG).
I have a couple of potential concerns here.
First, I think it's very useful to be able to test the conda packaging infrastructure and resulting packages for development versions. As I think you've explained, conda-forge only builds packages for release versions of a package, which means that we don't get to test the builds until after we've made a Cantera release, and any fixes that need to be made in Cantera either need to be handled as build-time patches (always tedious) or as a subsequent minor release of Cantera.
I also think it's useful for us to be able to distribute alpha/beta conda packages for testing purposes, and for users that like to stay closer to the cutting edge without always having to compile Cantera themselves.
Last, I thought that packages from the defaults channel and conda-forge weren't supposed to be mixed, or at least were likely to lead to lots of version conflicts. So if we don't build for the defaults channel, then we won't have a Cantera package available for users of mainline Anaconda (unless the PyPI package suffice here?).
Perhaps the first two of these could be mitigated by running CI ourselves on the recipe, and uploading those packages to a different conda channel?
It would also eliminate this class of problem, as long as the recipe was kept up-to-date as new dependency libraries are released. This is generally handled by automated pull requests. For example, Cantera would have to support the newest, say, sundials when it came out I think.
At least in the case of Sundials, that will often require a new Cantera release. I guess I'm not sure what the state of this is right now for the conda-forge packages and Cantera 2.6.
To Ingmar's comment:
The other issue is that Intel mac’s are being replaced by Apple Silicon, which also supports a move towards CF.
GH Actions runners for macOS arm64 are on the way, expected by the end of this year (see here, though I'm not sure I want to build for "Apply silicone" 🤣 ). For this release, I'm wondering if it's worth just building those packages locally and uploading them, given that my development machine is an M1 anyway.
from conda-recipes.
So what happens when a new version of the dependency is released? They trigger new builds of all dependent packages?
Yes, see conda-forge/cantera-feedstock#26 as one example
At least in the case of Sundials, that will often require a new Cantera release. I guess I'm not sure what the state of this is right now for the conda-forge packages and Cantera 2.6.
At least for now, it's pinned to sundials ==5.7: https://github.com/conda-forge/cantera-feedstock/blob/main/recipe/meta.yaml#L39 I guess that might limit versions of other dependencies, but I'm not sure what other packages depend on sundials to do a reverse dependency check.
conda-forge only builds packages for release versions of a package,
It should be possible to build versions while we're testing updates for an upcoming release. I think the policy is that they only want released versions uploaded to the Anaconda servers for distribution. For instance, merging to a feature branch prior to an upcoming release should be OK, I would think.
I also think it's useful for us to be able to distribute alpha/beta conda packages for testing purposes,
Yeah, we would lose this ability I suppose. We'd have to balance this against the maintenance needs of having our own recipe.
So if we don't build for the defaults channel, then we won't have a Cantera package available for users of mainline Anaconda (unless the PyPI package suffice here?).
This is also true, and the PyPI package would have the same problem potential set of problems that the conda forge package would have regarding compiler versions and ABI compatibility. There are about 140k downloads of 2.6.0 from conda forge and 100k from mainline Anaconda source. OTOH, there are 58.5M downloads of CF numpy and only 4.5M of mainline Anaconda source. I'm not sure that's helpful...
Perhaps the first two of these could be mitigated by running CI ourselves on the recipe, and uploading those packages to a different conda channel?
I think this could be possible, although one of the big benefits of CF is their automated setup of the infrastructure required for all these builds, which we'd have to somehow duplicate for at least a subset of the build matrix.
So the number of distinct builds/packages is just the sum of the HDF5 and Python versions, not the product.
This is encouraging! 🎉
In the end, to me, it's a balance of maintenance cost vs. benefit. In my opinion, there'd be a bigger benefit from having you and Ingmar focus more on developing Cantera and less on packaging it. YMMV 😄 This would also possibly be a chance to post a message out on the UG asking for help maintaining our recipe. If it's important to people, they can step up.
from conda-recipes.
Related Issues (20)
- The generated conda package cannot run with glibc 2.12 HOT 2
- building for ARM architecture HOT 3
- Feature request: include C++ headers in conda HOT 6
- libcantera-devel fails on Windows HOT 33
- Cantera conda package forces downgrade of recent numpy versions HOT 7
- License for code contained in this repo? HOT 3
- Cantera-dev is outdated HOT 7
- Improve documentation of conda build process HOT 1
- Installing Matlab interface via conda fails if MATLAB folder is not in standard location HOT 11
- Library importation bug in GCloud VM HOT 11
- Conda path replacement mangling Cantera data directories HOT 2
- `Conda Install` Development Cantera Version 3.0.0a2 HOT 12
- conda-forge update HOT 2
- conda-build mangles libcantera-devel build system files HOT 10
- Windows conda packages default to 2.5.1 HOT 11
- Update to handle introduction of Python extension support HOT 8
- Intel MKL HOT 1
- Broken packaging action for 3.1.0a2 HOT 4
- Add script to auto-generate build matrix HOT 4
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from conda-recipes.