Add constraints from julia process into Conda version resolution (somehow?) about condapkg.jl HOT 9 CLOSED

OK I've done some digging and thinking.

The constraint really is on libstdc++: SciPy 1.9 requires libstdc++ 12 but Julia 1.7 is built with an earlier libstdc++ (v11 I believe, looking at https://gcc.gnu.org/onlinedocs/gcc-12.1.0/libstdc++/manual/manual/abi.html).

So one solution is to have the Conda dependency libstdcxx-ng<12 in PythonCall (you can verify yourself that this will restrict to SciPy 1.8). But some issues with this are:

• The actual constraint depends on the version of Julia and the platform. How do we work this out reliably (without it becoming a burden to maintain)? Can we get the libstdc++ version Julia was build with from within Julia? Or is there an external source of this information?
• It will install a package (libstdcxx-ng) that is unnecessary for some users. Not a big problem.
• CondaPkg dependencies are declared statically, so cannot depend on the platform. One solution is to add a build script to PythonCall which adds the dependency. Or we just have a single dependency covering all supported versions of Julia.

from condapkg.jl.

Thanks that's useful feedback. I'll continue to mull it over.

As for making this configurable, I bet the intersection between people using custom Julia builds, people using PythonCall, and people needing a more recent libstdc++ than this allows is quite small! So probably not pressing. I've been wanting to add a generic way for the user to over-ride CondaPkg dependencies somehow, which would solve this generically without adding more config to PythonCall.

Tangentially, how well does W&B work from PythonCall? I assume basic metric logging works OK, but what about other "magic" stuff like logging the command, the script, the git commit, process-level stuff, etc.?

from condapkg.jl.

AFAIU Julia already has some libraries linked when it starts up, and these can shadow ones from Python packages, hence the incompatibility.

There's probably something we can do, but the hard part will be figuring out what the constraints should even be - I imagine they can change across Julia versions and operating systems.

Does the incompatibility really lie with SciPy or some dependency? What is the diff between the packages that are installed in the two configurations above?

from condapkg.jl.

I am pretty sure it is scipy because v1.9 added HiGHs (https://docs.scipy.org/doc/scipy/release.1.9.0.html#scipy-optimize-improvements) which is a C++ optimization solver that shows up in the error message above.

from condapkg.jl.

Can we get the libstdc++ version Julia was build with from within Julia? Or is there an external source of this information?

I asked about this a bit on the #binarybuilder channel on the Julia Slack. There @Keno suggested inspecting the version of the CompilerSupportLibraries_jll stdlib from within Julia and having a handwritten lookup table of CompilerSupportLibraries_jll versions to GCC versions.

I think the way to get the version from within Julia would be

julia> using TOML, CompilerSupportLibraries_jll

julia> TOML.parsefile(joinpath(pkgdir(CompilerSupportLibraries_jll), "Project.toml"))["version"]
"0.5.2+0"

I am not really sure how to build the lookup table though. CompilerSupportLibraries_jll itself is built here: https://github.com/JuliaPackaging/Yggdrasil/blob/master/C/CompilerSupportLibraries/build_tarballs.jl. And that script has the version number of CompilerSupportLibraries_jll that it is building. But the GCC version isn't similarly hardcoded, so my best guess is one needs to look at when the version number was bumped and try to figure out what GCC version was being used at that time. That seems very manual and error prone though.

CondaPkg dependencies are declared statically, so cannot depend on the platform. One solution is to add a build script to PythonCall which adds the dependency. Or we just have a single dependency covering all supported versions of Julia.

This seems tricky. I guess one starting point could just be clear documentation explaining the issue and suggesting adding a libstdcxx-ng compat bound (ideally with some information on how to figure out what the bound should be), without doing anything automatic.

from condapkg.jl.

I added a (secret for now) feature to CondaPkg to allow run-time dependencies, so PythonCall can now depend on libstdcxx-ng depending on the platform and version: https://github.com/cjdoris/PythonCall.jl/compare/main..cxx-compat

You can try it out on the cxx-compat branch. It's not too hard to work out the compatibility for each version of Julia, so I did that, but it's manual (grep some info, look up a web page...) so I just need to remember to update it whenever a new version of Julia is out.

Using the version of CompilerSupportLibraries_jll is a good idea, in that it should change no more often than the Julia version.

I'm still not 100% on including this functionality, but it does seem necessary since the libstdc++ version really is fixed by Julia. Any thoughts?

from condapkg.jl.

I think something like this would be really useful; it took me awhile to figure out why something was broken, and the error doesn't always surface in the nicest way.

E.g. for me it came up as the wandb python library depends on sklearn if you want to log ROC curves (so it's kinda an optional dep), and the error surfaced to me was "Install sklearn if you want to use ROC curves". Well I thought I was installing sklearn and it took some debugging to reproduce (e.g. only showed up on linux, not my mac laptop) and dig in to figure out actually sklearn was being installed but it was erroring at runtime because of it's dependency on scipy and this clash, and the wandb was swallowing the error.

So in other words, this kind of error is platform dependent so it can be hard to repro, and can show up unexpectedly, so I think any safety CondaPkg can add around this will be really helpful.

In terms of the implementation, I think it could be nice if it was configurable somehow, because I could see folks w custom Julia builds being annoyed, but I imagine 99% of users are using a stock julia build and adding the compat is good. So anyway maybe it's ok to assume stock for now and see if anyone complains 😄.

from condapkg.jl.

Tangentially, how well does W&B work from PythonCall? I assume basic metric logging works OK, but what about other "magic" stuff like logging the command, the script, the git commit, process-level stuff, etc.?

I mostly just use the metric logging, basic metrics but also plots and such. The command gets logged out as <python with no main file> but it does have the command-line arguments for the julia process. The git commit seems to work fine. It also tracks stuff like CPU usage, memory, gpu usage etc, no problem. I haven't tried the source code tracking stuff. We already had a pretty robust system for reproducibility/versioning/etc so for me the main purpose is logging and plots and reports to compare training runs, and for that it's been working fine. I would say for me the biggest annoyance of W&B-in-julia is the python dependency itself (which PythonCall/CondaPkg help a lot with!) just due to install issues like this, and slowing down docker builds etc.

from condapkg.jl.

I have just released a version of PythonCall with this fix in, so closing the issue.

It's cool that W&B works so well from Julia! I'm quite amazed actually.

from condapkg.jl.