can't import modules from virtualenv about libpython-clj HOT 25 CLOSED

@ksseono, @alanmarazzi - I am closing this issue with wontfix for now. I really do appreciate this thread and the work done to try to make things work. Environment issues are hard and extremely prevalent with python systems hence the explosion of things like conda, virtualenv, etc. Relatively more of your environment is controlled when using JVM languages but we can't change python.

Using the docker container is possible as is using my nextjournal entry. So, I want to redirect the problem to making clojure development from a docker container easier or better. This direction makes using advanced tech like tvm feasable and limits to some effect dev/production differences.

For instance in a previous project I was able to create a docker container that mounted the local project directory, used my /home/.m2, and ran as my userid (not root) and thus didn't make a mess in my projects. This was enough to drastically lowered the pain of using custom docker containers for development.

In the short run, I think a docker container or nextjournal solves the problem better than we can with this library at this time.

Please feel free to continue the discussion; it has been helpful already to me.

from libpython-clj.

Completely agree this is very important. Thanks for the issue, I think this is a great next direction to work towards. On that note, are there more things like virtualenv that we should be talking about? I thought Conda could also setup virtual environments.

from libpython-clj.

Thanks, @cnuernber. As you said, Conda should be an important option.
And how about pyenv? It could be little bit more tricky to deal with it comparing to do only with virtualenv, but I think most of the python developers can have the benefit of it to manage and dynamically choose various python versions even including virtualenvs through pyenv-virtualenv.

Do you have an idea where to start for this supports? Cause I want to try to contribute for this, but I'm not familiar with the internal process of this project yet ;)

from libpython-clj.

I have been thinking about this and there are a few things going on here.

From within your virtualenv, what version of python are you loading? This is one thing I am curious about; for instance is your keras compiled against python 3.5 and libpython is currently somewhat hardcoded to 3.6 this could be a cause of a hard crash.

1. In an ideal world nothing should crash the repl. So if it is the case where something was nil and that caused a hard crash then the libpython system should be fixed so that case is handled and an exception should be thrown. We can't stop someone calling exit(1) or something like that in some C code; I think numpy does this if it detects a version conflict but I am not convinced that we can't detect this condition and just have an exception instead of a hard crash.

2. I did research into virtualenv and it appears that it works via environment variables. Did you run source venv/bin/activate before starting the java process? From there I was hoping the c code would respect all the variables and off we go. It could be that the initial dynamic library loading system needs to find the shared library via the environment variables although I would think jna would handle that (wouldn't those virtual environments just reset the PATH environment variable?).

In fact, I wonder if Conda, pyenv-virtualenv, all of those all just work via the same mechanism. Researching into how JEP handles this may provide some insights quickly but it would be instructive to have a markdown document that lists the different virtualization systems and the means by which they work along with, for instance, actual details like the list of environment variables set by the different environments along with their meaning if they are obtuse or confusing.

I think great first moves could be:

• Research into how/if JEP handles python virtualization systems
• A document of what we know about the different python virtualization systems. This is probably a good move for the long run.
• Trying to detect and stop the hard crash
• Trying to get your specific instance to work by any means possible

Potentially the actual fix could be as simple as making sure the environment variables are setup before the java process starts and making sure the jna dynamic library loader respects those environment variables; if a full path is set here this will bypass the dynamic search mechanism.

As far as contributions go, it is up to us to decide how we want this project to work. For this to be as successful as I would like it to be we will need lots of help and support from lots of people so something simple, quick, and fair would be ideal at first. PR's work for me just fine for now especially if accompanied by tests. The longer the PR in terms of code the more back and forth will probably happen so shortest is always best.

This is a great issue I am really excited that you are interested in this!

from libpython-clj.

I just tested this with python -m venv and it worked fine assuming the environment was activated.

What was your pathway for testing this? Are you still seeing crashes?

from libpython-clj.

Thanks @cnuernber for your thoughtful answers!

I was able to execute the sample code, run-simple-string. I had to install python 3.6 to run libpython cause my initial python version was 3.7. For the keras-simple, I setup the virtualenv on the libpython-clj project directory and the environment was activated.

I 'assumed' the environment should be activated on the same directory where the project to be started, but if this is wrong or I missed something, please let me know. I need to figure it out what's the exact cause of this hard crash as you said.

from libpython-clj.

Could you try setting up the system to load the python3.7 libpython once the system is activated?

I think keras and such are compiled against particular versions of python and while I can't exactly see why anything should crash from the code (the C object sizes appear the same) it is super likely that everything needs to load the same version of python.

You can set a different version of the python library to load by doing

(alter-var-root! #'libpython-clj.jna.base/*python-library* (constantly "python3.7m"))

I have checked in a commented out version of this into the keras-simple example just before initialize! is called. If you have a moment, would you mind activating your environment, uncommenting this line and trying out the example?

from libpython-clj.

I tried the way you mentioned but unfortunately I couldn't make it.

1. If I try with python3.7m set up before initialize! then it's failing to load library even I created a symbolic link of libpython3.7m.dylib file to my system $PATH (yes, I'm testing on macOS Mojave, v10.14.5).

2. I've found that it seems like the libpython-clj is trying to load global modules instead of modules installed in virtualenv, following the version of libpython-clj.jna.base/*python-library*.

3. The repl is crashed when the module is not installed. For example, if I install numpy in global python env then it's properly working, but if I try without the installation of numpy then repl is unexpectedly closed (tested with python3.6m).

from libpython-clj.

OK, that is helpful but we are stuck. I didn't try without installing numpy itself, I tried without installing keras. I will try without numpy soon.

We need some python expert to help us out here and explain what is going on. I don't have a mojave based laptop to work on this on. One thing for sure is that you should be able to load a python3.7 version of the library somehow. That is extremely odd that you cannot and that is the version of python your system is running.

from libpython-clj.

Ah, I should've explained more. The reason why I tested with numpy is that it's just the same issue with keras, so I had wanted to test with the other.

I don't know your developing environment but to reproduce my issue, you could uninstall keras or numpy from your environment and run the project again. Because I can successfully load the libraries with the modules installed in global environment of python3.6.

In the mean while, I need to try to load python3.7 libraries ;)

from libpython-clj.

I uninstalled keras and got an exception. I didn't try numpy though and that could be different.

Another possible pathway is to grep the JEP codebase for venv or virtualenv and see if anything turns up.

from libpython-clj.

@ksseono where do you start the Clojure repl?

from libpython-clj.

@alanmarazzi From the root of my project, of course.

from libpython-clj.

Before or after activating the environment? And do you do it directly from the same bash session or from an editor or IDE?

I was able to make everything work, but on conda

from libpython-clj.

@alanmarazzi Sorry for my late response. I had tried in my emacs before and I've just run REPL in my terminal from the root of the project with python 3.6.8 activated, but I couldn't have made it. :(

But I've noticed that an error log is raised as follows:

:tech.resource.gc Reference thread starting
info: executing python initialize!
Library python3.6m found at [:system "python3.6m"]
#error {
 :cause ModuleNotFoundError: No module named 'numpy'

 :data {}
 :via
 [{:type clojure.lang.Compiler$CompilerException
   :message Syntax error compiling at (keras_simple/core.clj:18:1).
   :data #:clojure.error{:phase :compile-syntax-check, :line 18, :column 1, :source keras_simple/core.clj}
   :at [clojure.lang.Compiler load Compiler.java 7648]}
  {:type clojure.lang.ExceptionInfo
   :message ModuleNotFoundError: No module named 'numpy'

   :data {}
   :at [libpython_clj.python.interpreter$check_error_throw invokeStatic interpreter.clj 260]}]
 :trace
 [[libpython_clj.python.interpreter$check_error_throw invokeStatic interpreter.clj 260]
  [libpython_clj.python.interpreter$check_error_throw invoke interpreter.clj 257]
  [libpython_clj.python.object$wrap_pyobject invokeStatic object.clj 106]
  [libpython_clj.python.object$wrap_pyobject invoke object.clj 100]
  [libpython_clj.python.interop$import_module$fn__21974 invoke interop.clj 72]
  [libpython_clj.python.interpreter$with_gil_fn$fn__19705 invoke interpreter.clj 197]
  [clojure.lang.AFn applyToHelper AFn.java 152]
  [clojure.lang.AFn applyTo AFn.java 144]
  [clojure.core$apply invokeStatic core.clj 665]
  [clojure.core$with_bindings_STAR_ invokeStatic core.clj 1973]
  [clojure.core$with_bindings_STAR_ doInvoke core.clj 1973]
  [clojure.lang.RestFn invoke RestFn.java 425]
  [libpython_clj.python.interpreter$with_gil_fn invokeStatic interpreter.clj 195]
  [libpython_clj.python.interpreter$with_gil_fn invoke interpreter.clj 184]
  [libpython_clj.python.interop$import_module invokeStatic interop.clj 71]
  [libpython_clj.python.interop$import_module invoke interop.clj 69]
  [libpython_clj.python$import_module invokeStatic python.clj 69]
  [libpython_clj.python$import_module invoke python.clj 66]
  [keras_simple.core$eval22601 invokeStatic core.clj 18]
  [keras_simple.core$eval22601 invoke core.clj 18]
  [clojure.lang.Compiler eval Compiler.java 7177]
  [clojure.lang.Compiler load Compiler.java 7636]
  [clojure.lang.RT loadResourceScript RT.java 381]
  [clojure.lang.RT loadResourceScript RT.java 372]
  [clojure.lang.RT load RT.java 459]
  [clojure.lang.RT load RT.java 424]
  [clojure.core$load$fn__6841 invoke core.clj 6126]
  [clojure.core$load invokeStatic core.clj 6125]
  [clojure.core$load doInvoke core.clj 6109]
  [clojure.lang.RestFn invoke RestFn.java 408]
  [clojure.core$load_one invokeStatic core.clj 5908]
  [clojure.core$load_one invoke core.clj 5903]
  [clojure.core$load_lib$fn__6782 invoke core.clj 5948]
  [clojure.core$load_lib invokeStatic core.clj 5947]
  [clojure.core$load_lib doInvoke core.clj 5928]
  [clojure.lang.RestFn applyTo RestFn.java 142]
  [clojure.core$apply invokeStatic core.clj 667]
  [clojure.core$load_libs invokeStatic core.clj 5985]
  [clojure.core$load_libs doInvoke core.clj 5969]
  [clojure.lang.RestFn applyTo RestFn.java 137]
  [clojure.core$apply invokeStatic core.clj 667]
  [clojure.core$require invokeStatic core.clj 6007]
  [clojure.core$require doInvoke core.clj 6007]
  [clojure.lang.RestFn invoke RestFn.java 408]
  [user$eval5 invokeStatic form-init17977830527275239055.clj 1]
  [user$eval5 invoke form-init17977830527275239055.clj 1]
  [clojure.lang.Compiler eval Compiler.java 7177]
  [clojure.lang.Compiler eval Compiler.java 7166]
  [clojure.lang.Compiler eval Compiler.java 7166]
  [clojure.lang.Compiler load Compiler.java 7636]
  [clojure.lang.Compiler loadFile Compiler.java 7574]
  [clojure.main$load_script invokeStatic main.clj 475]
  [clojure.main$init_opt invokeStatic main.clj 477]
  [clojure.main$init_opt invoke main.clj 477]
  [clojure.main$initialize invokeStatic main.clj 508]
  [clojure.main$null_opt invokeStatic main.clj 542]
  [clojure.main$null_opt invoke main.clj 539]
  [clojure.main$main invokeStatic main.clj 665]
  [clojure.main$main doInvoke main.clj 618]
  [clojure.lang.RestFn applyTo RestFn.java 137]
  [clojure.lang.Var applyTo Var.java 705]
  [clojure.main main main.java 40]]}
nREPL server started on port 51692 on host localhost - nrepl://localhost:51692

It seems like it's trying to load numpy as keras-simple.core gives the command: (defonce np (import-module "numpy")).

What are your environments and settings to run?

from libpython-clj.

Do you have numpy in your env? Anyway I create a conda env, source activate it and then lein repl from the same session. I had issues with the Python version this way, but not if I have Python 3.6 and required libraries

from libpython-clj.

@alanmarazzi I'm using python 3.6 with pyenv, and of course I had installed numpy, keras and tensorflow. I'd also tested on python 3.7 env but faced the same result.

from libpython-clj.

Can you try with conda? If you don't want a bloated Anaconda install I suggest https://docs.conda.io/en/latest/miniconda.html. This would be really helpful to better understand what's going on (I'm very interested in this).

By the way, if in the meantime you want to play around there are Docker images for both libpython-clj and panthera depending on your use case

from libpython-clj.

@alanmarazzi, thank you so much for your tip! I'll try sooner or later and share the result here :)

from libpython-clj.

I see this is marked wontfix, so I just want to confirm that current thinking is that virtualenv and pyenv are not supported, but you can use docker and install python packages into the global python inside the docker container?

from libpython-clj.

@johntwill - This is wontfix because we don't have a way to fix this.

virtualenv and pyenv are somewhat supported.

Please see this issue and this issue.

Specifically, those environments don't always create a dylib. So you may have to reinstall your env with some environment variables indicating that you want it to create a shared library for python.

Then you can specify the executable path to the virtual env install of python created and the system will do its best to respect that path.

from libpython-clj.

Thank you for the reply! I like to use pyenv with direnv. I used that to create a pyenv virtualenv named cv-3.6.7, which was baselined from my pyenv python 3.6.7.

The steps I used to get it to work on ubuntu:

1. env PYTHON_CONFIGURE_OPTS="--enable-shared" pyenv install 3.6.7
2. use direnv to create a pyenv virtualenv cv-3.6.7.
3. export PYTHONHOME=~/.pyenv/versions/cv-3.6.7
4. In the code:(py/initialize! :python-executable "/home/x/.pyenv/versions/3.6.7/bin/python") before python-require

from libpython-clj.

This is probably too late but might help someone else to save some time.
I just followed the advice of @alanmarazzi and tried miniconda, after not having much success either with just pyenv and I got it finally working, with pyenv. I've never used conda with pyenv before so I'm not sure if this is a good solution, but it works.
You need the pyenv-venv plugin to handle virtual environments with pyenv https://github.com/pyenv/pyenv-virtualenv

I tested it with Carin Meiers example repo https://github.com/gigasquid/libpython-clj-examples.

Try:

• pyenv install miniconda3-latest
• pyenv virtualenv miniconda3-latest conda-nltk-clj
• pyenv activate miniconda3-latest/envs/conda-nltk-clj
• conda activate conda-nltk-clj
• conda init zsh (optional - i guess)
• pip install nltk
• navigate to the project root and start your repl

❯ lein repl
If there are a lot of uncached dependencies this might take a while ...
nREPL server started on port 51736 on host 127.0.0.1 - nrepl://127.0.0.1:51736
REPL-y 0.4.4, nREPL 0.7.0
Clojure 1.10.1
OpenJDK 64-Bit Server VM 11.0.8+10
    Docs: (doc function-name-here)
          (find-doc "part-of-name-here")
  Source: (source function-name-here)
 Javadoc: (javadoc java-object-or-class-here)
    Exit: Control+D or (exit) or (quit)
 Results: Stored in vars *1, *2, *3, an exception in *e

gigasquid.-configure=> (require 'gigasquid.nltk)
Jul 27, 2020 2:36:22 PM clojure.tools.logging$eval505$fn__508 invoke
INFO: Executing python initialize with options:{:python-executable nil, :program-name nil, :python-home nil, :library-path nil}
Jul 27, 2020 2:36:22 PM clojure.tools.logging$eval505$fn__508 invoke
INFO: Detecting startup-info for Python executable:
Jul 27, 2020 2:36:23 PM clojure.tools.logging$eval505$fn__508 invoke
INFO: Startup info detected:
{:lib-version "3.8",
 :java-library-path-addendum
 "/Users/smittie/.pyenv/versions/miniconda3-latest/envs/conda-nltk-clj/lib",
 :exec-prefix
 "/Users/smittie/.pyenv/versions/miniconda3-latest/envs/conda-nltk-clj",
 :executable
 "/Users/smittie/.pyenv/versions/miniconda3-latest/envs/conda-nltk-clj/bin/python3",
 :libnames ("python3.8m" "python3.8"),
 :prefix
 "/Users/smittie/.pyenv/versions/miniconda3-latest/envs/conda-nltk-clj",
 :base-prefix
 "/Users/smittie/.pyenv/versions/miniconda3-latest/envs/conda-nltk-clj",
 :base-exec-prefix
 "/Users/smittie/.pyenv/versions/miniconda3-latest/envs/conda-nltk-clj",
 :python-home
 "/Users/smittie/.pyenv/versions/miniconda3-latest/envs/conda-nltk-clj",
 :version [3 8 3],
 :platform "darwin"}

Jul 27, 2020 2:36:23 PM clojure.tools.logging$eval505$fn__508 invoke
INFO: Trying python library names ["python3.8m" "python3.8" "python3.7m" "python3.6m"]
Jul 27, 2020 2:36:23 PM clojure.tools.logging$eval505$fn__508 invoke
INFO: Setting java library path: /Users/smittie/.pyenv/versions/miniconda3-latest/envs/conda-nltk-clj/lib:/Users/smittie/Library/Java/Extensions:/Library/Java/Extensions:/Network/Library/Java/Extensions:/System/Library/Java/Extensions:/usr/lib/java:.
Jul 27, 2020 2:36:23 PM clojure.tools.logging$eval505$fn__508 invoke
INFO: Reference thread starting
Jul 27, 2020 2:36:23 PM clojure.tools.logging$eval505$fn__508 invoke
INFO: Library python3.8 found at [:java-library-path "/Users/smittie/.pyenv/versions/miniconda3-latest/envs/conda-nltk-clj/lib/libpython3.8.dylib"]
Jul 27, 2020 2:36:23 PM clojure.tools.logging$eval505$fn__508 invoke
INFO: Library c found at [:system "c"]
nil
gigasquid.-configure=> (ns gigasquid.nltk)
nil
gigasquid.nltk=> (require-python '([nltk.book :as book]))
:ok

I've tested it under OSX 10.15.6, just for the record.

from libpython-clj.

After a few unsuccessful attempts with venv and virtualenv, I attempted to follow @lockejan's instructions of using pyenv and the associated virtualenv plugin along with miniconda3-latest.

Assuming one has pyenv and the virtualenv plugin setup, the following seemed enough here to get things working. (Note that this is less than the original steps).

pyenv install miniconda3-latest
pyenv virtualenv miniconda3-latest virtualenv-name
pyenv activate miniconda3-latest/envs/virtualenv-name

Note in the last step it appears one can get:

Failed to activate virtualenv.

Perhaps pyenv-virtualenv has not been loaded into your shell properly.
Please restart current shell and try again.

In my case this appeared to be because I hadn't initialized pyenv in the shell I was using, and the following seemed to be enough:

eval "$(pyenv init -)"

(IIUC, this incantation is mentioned in the README of pyenv.)

Subsequent to this I started an appropriate Clojure/JVM process via clj and was able to verify via a socket repl that the comment block code in the libpython-clj template worked.

So first, thanks @lockejan for sharing your success 👍

For the record, the system I tested on was an Arch Linux derivative.

I will investigate a bit more as to why this works. My current suspicion is that it's because it doesn't try to build the virtual environment based on the operating-system-installed python. Perhaps vanilla venv and/or virtualenv can also work if they are instructed to do likewise. I mean, isn't venv or virtualenv being used via pyenv?

from libpython-clj.

Relying on a version of python that is installed (here I don't mean using something like NixOS or Guix, but rather more a traditional arrangement) on one's system seems like asking for trouble -- e.g. the OS has its own priorities so who knows what will happen after an update, or perhaps one needs to test against different versions of python, etc.

pyenv + virtualenv seems like it might work for Linux/macos and IIUC there is a similar pyenv-win project so may be that's an option.

So along those lines...had some luck getting things to work with pyenv + virtualenv without miniconda.

Some digging turned up this text:

For instance, if the Python executable is found in /usr/local/bin/python, it will assume that the libraries are in /usr/local/lib/pythonX.Y. (In fact, this particular path is also the “fallback” location, used when no executable file named python is found along PATH.) The user can override this behavior by setting the environment variable PYTHONHOME, or insert additional directories in front of the standard path by setting PYTHONPATH.

via: https://docs.python.org/3/c-api/intro.html#embedding-python

According to the python man page:

  PYTHONHOME
         Change the location of the standard Python  libraries.   By  de‐
         fault,  the  libraries are searched in ${prefix}/lib/python<ver‐
         sion> and  ${exec_prefix}/lib/python<version>,  where  ${prefix}
         and  ${exec_prefix} are installation-dependent directories, both
         defaulting to /usr/local.  When $PYTHONHOME is set to  a  single
         directory, its value replaces both ${prefix} and ${exec_prefix}.
         To specify different values for these, set $PYTHONHOME to ${pre‐
         fix}:${exec_prefix}.

  PYTHONPATH
         Augments  the  default search path for module files.  The format
         is the same as the shell's $PATH: one or  more  directory  path‐
         names   separated   by  colons.   Non-existent  directories  are
         silently ignored.  The default search path is  installation  de‐
         pendent, but generally begins with ${prefix}/lib/python<version>
         (see PYTHONHOME above).  The default search path is  always  ap‐
         pended  to  $PYTHONPATH.  If a script argument is given, the di‐
         rectory containing the script is inserted in the path  in  front
         of  $PYTHONPATH.  The search path can be manipulated from within
         a Python program as the variable sys.path.

With this info in hand, the following worked here (Linux) assuming pyenv and pyenv-virtualenv (its plugin for virtualenv) are installed:

1. Use pyenv to install some version of python (e.g. 3.8.5):

pyenv install 3.8.5

This is used as a base for a variety of virtualenvs later. Installing a non-system-tied version of python here helps to reduce potential breakage from system-related changes.

2. Prepare a project directory:

mkdir my-project

3. Within the project directory, indicate which version of python to use for the project:

cd my-project && pyenv local 3.8.5

4. Create a virtualenv (presumably one can make others later if necessary) for the project:

pyenv virtualenv 3.8.5 my-project-virtualenv-1

5. Activate the virtualenv:

pyenv activate my-project-virtualenv-1

If one skips this step, dependencies installed (e.g. via pip in the next step), may end up in the base python directory. If one works with multiple projects, this is probably not desired.

6. Install any dependencies / modules / libraries, e.g. tree_sitter:

pip install tree_sitter

The "installed" piece should end up in the virtualenv, keeping the base python installation "fresh" and resuable for other projects or virtualenvs.

7. Appropriately set PYTHONHOME and PYTHONPATH:

PYTHONHOME=$HOME/.pyenv/versions/3.8.5
PYTHONPATH=$HOME/.pyenv/versions/my-project-virtualenv-1/lib/python3.8/site-packages

IIUC, setting PYTHONHOME is about ensuring that python knows where its standard libraries are. I think tweaking this may help address this issue: #96 (comment).

IIUC, setting PYTHONPATH to point at a part within the virtualenv may help in ensuring that python can find additional libraries (e.g. those installed in step 6).

With these preparations in place, had some success in getting a project based on a slightly updated libpython-clj template to work, including interop access of a python library installed at step 6.

from libpython-clj.