Any way to indicate pip arch? about condapkg.jl HOT 13 CLOSED

I'm definitely no expert in this stuff, but in order to use a package of a different arch, do you need to run Python itself on the same arch? And therefore run Julia on that arch if you're using PythonCall?

from condapkg.jl.

No, I don't think so. Python and/or macOS seems to be able to figure it out. I have ARM Python installed and some x86 and some ARM pip packages working through the same install. (I presume the binaries are translated by the OS via Rosetta behind the scenes?)

from condapkg.jl.

What is the output of the following two commands?

python -c "import distutils; print(distutils.util.get_platform().replace('-','_'))"
arch -x86_64 python -c "import distutils; print(distutils.util.get_platform().replace('-','_'))"

I'm wondering if we can use the --platform argument to pip to achieve the same thing. Does the following command work, where P1 and P2 are the two platforms printed above?

pip install --platform P1 --platform P2 casatools

from condapkg.jl.

Ah, I see I was mistaken. The output for both time for the above commands was macosx_10.9_x86_6.

If I go to my .CondaPkg/env/bin/ directory and run the commands there, I get macosx_11.0_arm64 and arch: posix_spawnp: ./python: Bad CPU type in executable, respectively.

If I run the following (in the .CondaPkg/env/bin/ directory):
./pip install --platform macosx_11.0_arm64 --platform macosx_11.0_x86_64 casatools
I get the error:
ERROR: When restricting platform and interpreter constraints using --python-version, --platform, --abi, or --implementation, either --no-deps must be set, or --only-binary=:all: must be set and --no-binary must not be set (or must be set to :none:).

If I run:
./pip install --only-binary=:all: --platform=macosx_11_0_x86_64 --target=/Users/bensett/.julia/env/v1.8/ casatools
I get a successfully installed message, but then when I try to import casatools from the CondaPkg python, I get a module not found error.

So it seems that I really don't know what I am doing here, but if it is possible to specify the arch when specifying the python version in CondaPkg.toml, perhaps that might help get some headway?

from condapkg.jl.

Since I don't have an ARM Mac I can't do anything until you can show me a reliable method to install the package under a different arch - then I can replicate the process in CondaPkg.

So I'd recommend creating a fresh Conda environment containing Python (e.g. conda create -n test -c conda-forge test), activate it (conda activate test), check the platform is arm64 using the command above, and then figure out how to install a working x64 version of casatools into the environment, such as by playing around with the --platform option to pip.

from condapkg.jl.

I just understood what you meant by "if it is possible to specify the arch when specifying the python version in CondaPkg.toml, perhaps that might help get some headway". You can set CONDA_SUBDIR=osx-64 yourself to set the platform - can you give it a try? You may need to delete any existing Conda environment.

from condapkg.jl.

Could you also please try installing an x86_64 version of Julia and seeing if that works?

from condapkg.jl.

until you can show me a reliable method to install the package under a different arch

I have not been successful yet in accomplishing this.

You can set CONDA_SUBDIR=osx-64 yourself to set the platform - can you give it a try?

I deleted my .CondaPkg directory, launched Julia and set ENV["CONDA_SUBDIR"] = "osx-64" in before running ]conda run python --version. If I look at the new .CondaPkg/env/bin/python that was subsequently installed, it is now macosx_10.9_x86_64!

If I would like to use this solution in my Julia package, I assume I would need to include a statement like:

    if Sys.isapple() && Sys.ARCH == :aarch64
        ENV["CONDA_SUBDIR"] = "osx-64" # Use x86_64 Python
    end

somewhere in my module? Where is the best place to do this so that when I use PythonCall, it grabs the correct version of Python?

Could you also please try installing an x86_64 version of Julia and seeing if that works?

I tried this and it also works, though it is not my preferred solution. (Worked out of the box that is, without setting ENV. For this install of Julia, Sys.ARCH == :x86_64)

~~~

For reference, my MWE CondaPkg.toml file looks like:

[deps]
python = ">=3.8, <3.9"

[pip.deps]
casatools = ""

from condapkg.jl.

Ok great! Can you actually import casatools using PythonCall with either of these approaches?

from condapkg.jl.

Hmm, no. It seems I cannot launch PythonCall anymore.

julia> using PythonCall
┌ Warning: Python library "/Users/bensett/.julia/environments/v1.8/.CondaPkg/env/lib/libpython3.8.dylib" could not be opened.
└ @ PythonCall.C ~/.julia/packages/PythonCall/3GRYN/src/cpython/context.jl:117
ERROR: InitError: ArgumentError: NULL library handle
Stacktrace:
  [1] #dlsym#1
    @ ./libdl.jl:57 [inlined]
  [2] dlsym
    @ ./libdl.jl:56 [inlined]
  [3] init_pointers(p::PythonCall.C.CAPIPointers, lib::Ptr{Nothing}) (repeats 2 times)
    @ PythonCall.C ~/.julia/packages/PythonCall/3GRYN/src/cpython/pointers.jl:284
  [4] init_context()
    @ PythonCall.C ~/.julia/packages/PythonCall/3GRYN/src/cpython/context.jl:137
  [5] __init__()
    @ PythonCall.C ~/.julia/packages/PythonCall/3GRYN/src/cpython/CPython.jl:21
  [6] _include_from_serialized(pkg::Base.PkgId, path::String, depmods::Vector{Any})
    @ Base ./loading.jl:831
  [7] _require_search_from_serialized(pkg::Base.PkgId, sourcepath::String, build_id::UInt64)
    @ Base ./loading.jl:1039
  [8] _require(pkg::Base.PkgId)
    @ Base ./loading.jl:1315
  [9] _require_prelocked(uuidkey::Base.PkgId)
    @ Base ./loading.jl:1200
 [10] macro expansion
    @ ./loading.jl:1180 [inlined]
 [11] macro expansion
    @ ./lock.jl:223 [inlined]
 [12] require(into::Module, mod::Symbol)
    @ Base ./loading.jl:1144
during initialization of module C

If I delete the .CondaPkg directory, remove the casatools dependency from CondaPkg.toml, restart Julia, and install python without changing the arch (so letting it default to arm64), I can start PythonCall without issue.

Note that for my Julia install that was for x86_64 from the start, I have no problem starting PythonCall and importing casatools

julia> Sys.ARCH
:x86_64

julia> using PythonCall

julia> casatools = pyimport("casatools")
Python module: <module 'casatools' from '/Users/bensett/.julia/environments/v1.7/.CondaPkg/env/lib/python3.8/site-packages/casatools/__init__.py'>

julia> casatools.version()
Python list: [6, 5, 1, 23]

from condapkg.jl.

Hmm ok, I suspect that using x86_64 Julia is the only way to this is going to work - unless you can find a way to load an x86_64 package in arm64 Python.

If you want this to work in arm64 Julia, your time is probably better spent just helping casatools to make an arm64 build.

from condapkg.jl.

Yeah, I was afraid of that. I've already asked the devs there if they'd be willing to maintain a pip wheel (or an equivalent conda package) with support for ARM and they do not have the resources at this time to do that. (And I have neither the resources nor know-how to maintain this myself).

Thank you so much for your time looking into this! The x86_64 Julia route will have to do.

from condapkg.jl.

Sure no problem. On the plus side I'm glad to hear CondaPkg and PythonCall work without a hitch on x86_64 Julia on ARM Macs!

from condapkg.jl.