centralized environment about condapkg.jl HOT 12 CLOSED

I have a fairly heavyweight Python dependency provided by PyThermo.jl (~1.1 GB) that gets duplicated in every project/environment PyThermo's installed in. In most cases, there's only the one big Python dependency - it'd be great to keep Conda files in some centralized location akin to ~/.julia/packages. Maybe ~/.julia/conda/envs//, and cache by a content hash of .CondaPkg/meta?

from condapkg.jl.

Currently, you can create a Conda environment yourself, activate it, then load CondaPkg with JULIA_CONDAPKG_BACKEND=Current which will install any requirements into the environment you just made. You can put this environment wherever you like, and nothing else will get put into your Julia depot.

What exactly do you need that this doesn't do?

from condapkg.jl.

Currently, you can create a Conda environment yourself, activate it

I'd wish CondaPkg to manage this part by using JULIA_CONDAPKG_BACKEND=MicroMamba, without manual intervention, and being self contained.

julia> using CondaPkg
julia> conda_prefix() = "..."
julia> get!(ENV, "MAMBA_ROOT_PREFIX", conda_prefix())
julia> get!(ENV, "JULIA_CONDAPKG_TOPENV", conda_prefix())
julia> get!(ENV, "JULIA_CONDAPKG_BACKEND", "MicroMamba")
julia> CondaPkg.resolve()
julia> CondaPkg.add("matplotlib")

I also want to point out that it is impossible to currently reuse the MicroMamba installs using the CONDA_PREFIX and JULIA_CONDAPKG_BACKEND=Current approach since no command such as micromamba, mamba or conda is installed.

Assuming that any of the command is available in PATH is imo an invalid assumption (invalid in my centralized repo - dedicated to julia use only - case):
https://github.com/cjdoris/CondaPkg.jl/blob/c07efdfc67e5547df9a1e1309172f127a0649d6f/src/backend.jl#L14-L22

This would also be a flaw if CondaPkg picks a random conda (or alternative) command in the PATH. I work with multiple conda environments daily, and I absolutely have no wish for CondaPkg to mess up with my global installations.

The approach using JULIA_CONDAPKG_TOPENV worked for me.

from condapkg.jl.

@t-bltg The Current backend is not really intended for the way you are using it. It is aimed at someone who has manually created an environment themselves at the command line, so it is reasonable to assume that the conda (or mamba or micromamba) command is in the PATH in this case.

Anyway, that's by the by. I think what you really want is a means to specify where the environment is put, say JULIA_CONDAPKG_ENV=/path/to/env or JULIA_CONDAPKG_ENV=@name. This would allow different projects to share Conda environments, which I think is what you're after?

from condapkg.jl.

@stillyslalom The Conda root directory (where packages are cached to) is in a central place in the Julia depot, so is shared by different projects. When packages are installed into different projects by Conda, the files are (by default) hardlinked. This should mean that very little extra disk space is actually used when you have the same package installed in several projects.

from condapkg.jl.

Interesting! I've seen CondaPkg downloading dependencies on using PyThermo in an environment where it's already installed - is that just an auto-update?

from condapkg.jl.

I think what you really want is a means to specify where the environment is put, say JULIA_CONDAPKG_ENV=/path/to/env or JULIA_CONDAPKG_ENV=@name. This would allow different projects to share Conda environments, which I think is what you're after?

Exactly.

from condapkg.jl.

Ok thanks, shouldn't be too hard to implement.

from condapkg.jl.

@stillyslalom Please open a new issue if you'd like to discuss further.

from condapkg.jl.

@cjdoris, just to be clear, do you expect me to work on this and adjust #53, or do you have any plans for implementing it yourself ?

from condapkg.jl.

I can do it, but may not have time before the new year. I'm happy to take a PR for it, but it may be reasonably involved to do it right.

The approach in #53 is appealingly simple, but will end up re-resolving the environment every time you run Julia in a different project because the dependencies will have changed.

I think the "right" way to do this is to still store the CondaPkg metadata in a per-project location, but allow the conda environment itself to be somewhere else (controlled by an env var), and when in this mode of operation, only ever install packages, never uninstall (because otherwise you may uninstall something required by another project).

from condapkg.jl.

I've updated #53 to use the approach you described, using per-project metadata makes sense.
It works well for my usage, and I've disallowed uninstall when a shared environment is detected.

Can you disable the waiting approval for the ci run ? It's cumbersome to request approval for each gh action run.

Side note: micromambas progress bar is annoying when running the test locally (julia closes stdin for the tests, hence micromamba cannot determine whether the terminal emulator is a tty and floods the terminal with progress bar characters): a workaround is to set CI=true.

from condapkg.jl.