Comments (11)
The best workaround I have so far:
- Activate your environment -
conda activate envtest
- Export python home to the config prefix -
export PYTHONHOME=$(python3-config --prefix)
3a. Export JVM_OPTS so leiningen picks them up:
JVM_OPTS="-Djava.library.path=$(python3-config --prefix)/lib" lein repl
3b. Update your shared library paths to similar to 3a.
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:"$(python3-config --prefix)/lib"
3c. Explicitly load the library:
user> (require '[libpython-clj.python :as py])
nil
user> (require '[clojure.java.shell :as sh])
nil
user> (sh/sh "python3" "--version")
{:exit 0, :out "", :err "Python 3.6.9 :: Anaconda, Inc.\n"}
user> (System/getenv "PYTHONPATH")
nil
user> (System/getenv "PYTHONHOME")
"/home/chrisn/.conda/envs/envtest"
user> (py/initialize! :library-path (format "%s/lib/libpython3.6m.so"
(System/getenv "PYTHONHOME")))
:ok
user> (py/import-module "numpy")
<module 'numpy' from '/home/chrisn/.conda/envs/envtest/lib/python3.6/site-packages/numpy/__init__.py'>
from libpython-clj.
Related:
from libpython-clj.
Conda now works out of the box in a test with no environment variable setting like above. Leaving issue open for a bit to get more feedback.
from libpython-clj.
I added support for conda-installing Clojupyter earlier this year. I'm not sure exactly what you'll be trying to accomplish, but some it might be useful. Even though I only use a few of the environment variables, I added access to all of them in env.clj.
-Klaus.
from libpython-clj.
@klausharbo - Great to hear from you!
What we needed to do was to find the python shared libraries when conda was activated and initialize them with enough information that we can load the same python modules and you would be able to were you to activate the environment and run the python REPL.
I am having a bit of trouble full comprehending that file you linked to. These are variables that you expect to find in your environment when conda installs clojupyter I guess as part of the install process?
from libpython-clj.
I was probably too fast in posting - sorry for being obtuse. :-) I did struggle quite a bit in
getting to grips with Conda, and I'm not sure I understand very well even now, but I can share what
I found.
First off, I found the Conda Deep Dive video helpful in explaining the fundamental concepts. There's quite a bit of Conda documentation, but I have found it hard to understand - it simply doesn't always seem to address the questions I have. It may very well be because I don't really know Python and so the finer points are lost on me.
My notes recapping some of the early slides in the video:
Package
index.json
- The source of truth for things like package name and version.
- Without this, it is not a conda package. Used to build
repodata.json
.
info/paths.json
- Per-path meta-data
info/link.json
- Install-time meta-data. Optional.
- Special instructions on how to install package
- Description of whole package, not per-path
info/repodata_record.json
- Not part of package
- Created immediately after conda extracts the package in the
package cache - Looks a lot like
index.json
but has extra information:
channel, MD5 of tar, origin URL
Channel
- Example:
https://repo.anaconda.org/pkgs/main
- A channel is a collection of packages
- Channels have 'subdirs'
- Groups of platform-specific packages:
linux-64
,osx-64
,win-64
noarch
is the universal and common subdir among all platforms- Used to detect that a channel is in fact a channel
- Each subdir has a
repodata.json
file: A manifest of a packages in
the subdir - New type of file:
channeldata.json
- Sits in channel directory, describes channel
- Channel URL
- scheme, auth, location, token, name, subdir, package\
filename
- scheme, auth, location, token, name, subdir, package\
Environment
- Based on (uses) UNIX FHS (File Hierarchy Standard)
conda-meta
directory: Defines that a directory is a conda environmentconda-meta/history
lists all operations performed in the environment since its creation- JSON file for every package in the environment, contains all metadata about the package
The obscure env.clj
I linked to above is based on conda-build
Environment Variables. On second thought I don't think it's very relevant here: Conda uses environment variables to control what's going on, but these have to do with building a package (like Clojupyter), whereas what I think you guys are trying to is use Python in a Conda-controlled context.
For that I suspect the key environment variable is CONDA_PREFIX
, which is set to point to the active environment.
In my setup right now (I recently switched to Miniconda since I don't need all things included in Anaconda):
(base) locke:~/lab/clojupyter
08:54 > conda info --envs
# conda environments:
#
/Users/klaush/anaconda3
base * /Users/klaush/miniconda3
env1 /Users/klaush/miniconda3/envs/env1
env2 /Users/klaush/miniconda3/envs/env2
(base) locke:~/lab/clojupyter
08:54 >
where we have
(base) locke:~/lab/clojupyter
08:54 > echo $CONDA_PREFIX
/Users/klaush/miniconda3
(base) locke:~/lab/clojupyter
08:55 >
but doing
08:55 > conda activate env1
(env1) locke:~/lab/clojupyter
08:56 >
means we get
08:56 > echo $CONDA_PREFIX
/Users/klaush/miniconda3/envs/env1
(env1) locke:~/lab/clojupyter
08:56 >
Depending on how much detail you need, I suspect (Edited: strikethrough)$CONDA_PREFIX/conda-meta/index.json
could be interesting (mentioned under Package above).
from libpython-clj.
It doesn't look like my last comment about $CONDA_PREFIX/conda-meta/index.json
is correct. index.json
is a package concept, not an environment one :-(.
from libpython-clj.
Your perspective is correct; we are working with the runtime environment, not attempting to build a package.
from libpython-clj.
Next problem:
:ok
libpython-clj.python> (def mxnet (import-module "mxnet"))
Execution error at libpython-clj.python.interpreter/check-error-throw (interpreter.clj:394).
Traceback (most recent call last):
File "/home/chrisn/.conda/envs/pyclj/lib/python3.6/site-packages/mxnet/__init__.py", line 24, in <module>
from .context import Context, current_context, cpu, gpu, cpu_pinned
File "/home/chrisn/.conda/envs/pyclj/lib/python3.6/site-packages/mxnet/context.py", line 24, in <module>
from .base import classproperty, with_metaclass, _MXClassPropertyMetaClass
File "/home/chrisn/.conda/envs/pyclj/lib/python3.6/site-packages/mxnet/base.py", line 213, in <module>
_LIB = _load_lib()
File "/home/chrisn/.conda/envs/pyclj/lib/python3.6/site-packages/mxnet/base.py", line 204, in _load_lib
lib = ctypes.CDLL(lib_path[0], ctypes.RTLD_LOCAL)
File "/home/chrisn/.conda/envs/pyclj/lib/python3.6/ctypes/__init__.py", line 348, in __init__
self._handle = _dlopen(self._name, mode)
OSError: libmkldnn.so.0: cannot open shared object file: No such file or directory
from libpython-clj.
Solved by using LD_LIBRARY_PATH in lein repl startup script.
from libpython-clj.
This (LD_LIBRARY_PATH) is as good as we can get:
For a working pathway, use the conda docker container.
from libpython-clj.
Related Issues (20)
- process hanging in embeded mode on exception HOT 2
- from-import does not support "string"
- newbie: I get: `dir already refers to: #'clojure.repl/dir in namespace: user` HOT 2
- Newbie: Lein Clojure Windows Anaconda VSCode Cava REPL Returns (NoClassDefFoundError) HOT 5
- Create Python Module then Import Module and Call Function Inside Clojure? HOT 4
- Fix ubuntu version in dockerfiles HOT 6
- Consider releasing returned values from instance fns
- JDK-20 support HOT 3
- builtins/eval throws 'frame does not exist' HOT 2
- Unable to require libraries from within docker using conda HOT 6
- Failed to find a valid python library on MS-Windows with the official python distributions
- Extend CI coverage to all the mainstream architectures
- require-python missing functions with annotation HOT 3
- Make polyglot development easier by allow require-python to be path-informed HOT 1
- Data transformation pipelines? HOT 1
- NSWindow main thread error when trying to setup Gymnasium (openapi gym) HOT 5
- JVM crashes when requiring `libpython-clj2.require` on 64bit Raspberry Pi 4 HOT 1
- error of unhashable type HOT 15
- Ways to reduce probability of memory leaks? HOT 2
- add .devcontainer setup
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from libpython-clj.