clj-python / libpython-clj Goto Github PK
View Code? Open in Web Editor NEWPython bindings for Clojure
License: Eclipse Public License 2.0
Python bindings for Clojure
License: Eclipse Public License 2.0
I can't load the library, running in a REPL.
12:13:00 ~/Workspaces/clojure/cljfx-play $
→ python --version
Python 3.8.0
12:13:02 ~/Workspaces/clojure/cljfx-play $
→ java --version
openjdk 11.0.6 2020-01-14
OpenJDK Runtime Environment AdoptOpenJDK (build 11.0.6+10)
OpenJDK 64-Bit Server VM AdoptOpenJDK (build 11.0.6+10, mixed mode)
12:14:10 ~/Workspaces/clojure/cljfx-play $
→ clj
Clojure 1.10.1
user=> (require '[libpython-clj.require :refer [require-python]])
Jan. 24, 2020 12:07:47 PM clojure.tools.logging$eval494$fn__497 invoke
INFO: Executing python initialize with options:{:python-executable nil, :program-name nil, :python-home nil, :library-path nil}
Jan. 24, 2020 12:07:47 PM clojure.tools.logging$eval494$fn__497 invoke
INFO: Detecting startup-info for Python executable:
Jan. 24, 2020 12:07:47 PM clojure.tools.logging$eval494$fn__497 invoke
INFO: Startup info detected: {:python-home "/Users/stuartrexking/.pyenv/versions/3.8.0", :lib-version "3.8", :libname "python3.8m", :java-library-path-addendum "/Users/stuartrexking/.pyenv/versions/3.8.0/lib"}
Jan. 24, 2020 12:07:47 PM clojure.tools.logging$eval494$fn__497 invoke
INFO: Setting java library path: /Users/stuartrexking/.pyenv/versions/3.8.0/lib:/Users/stuartrexking/Library/Java/Extensions:/Library/Java/Extensions:/Network/Library/Java/Extensions:/System/Library/Java/Extensions:/usr/lib/java:.
Jan. 24, 2020 12:07:47 PM clojure.tools.logging$eval494$fn__497 invoke
INFO: Reference thread starting
Syntax error (ExceptionInfo) compiling at (libpython_clj/metadata.clj:13:1).
Failed to load library
user=>
Per comment from community member it appears that require-python
's use of :all
might not match Clojure's use of :all
. i.e., it seems that the behavior of :all
keeps the interned symbols private. I think it's important (for the sake of library authors and symbol exporting) to be able to make those public but it should probably use a different syntax. If we're going to play in Clojure land we need to follow Clojure rules!
from gensim.models.keyedvectors import KeyedVectors
Clojure 1.10.1
user=> (ns user (:require [libpython-clj.python :as py]))
Execution error (NullPointerException) at tech.jna.base/fn (base.clj:35).
null
I'm use python 3.7 and compile project as lein uberjar.
Thanks.
@pywith pybuiltin("open")("file.txt","w") as f begin
f.write("hello")
end
The libpython system was built with incomplete support for multiple interpreters.
Filling out this support will allow one to control python from multiple threads and use some serializable communication format (like pickle or json) to communicate python objects between interpreters.
The rules for libpython w/r/t threading are fairly clearly written on their threading API page. Interpreters of some scope of memory allocated to them and their threading primitives include at least:
libpython-clj's behavoir w/r/t all this is somewhat summarized here.
The gist is this - libpython binds the current interpreter to a thread-local variable when it acquires the gil. If one is bound then it attempts a swap. It does this with a release on try...finally
. It also uses Java synchronization to make sure that interpreters are not reentrant in some unexpected way.
So, the work required here is to, using the libc bindings, build out the pathway further for multiple interpreters. A rough outline may be:
(defn create-interpreter ...)
My opinion is that 1,2 would be useful enough to be a clear PR. Then we can carry it forward with 3 and 4 in separate stages being careful to check and be correct as we can w/r/t memory and threading semantics at each step. I also believe this is an important and necessary in the growth of this library.
The relevant portions of the zulip chat that prompted this are pasted below.
I was thinking about it, but would it be possible to create multiple python interpreters/processes in a single/multiple namespaces? If yes, we could sell it as a solution for concurrency in python :) That being said it will be interesting to be able to provide explicitly an interpreter/python process to a module, or the ability to explicitly attach a module to an interpreter.
It would also means we could have isolation of data in the data analysis processes avoid namespace pollution all too common in python.
(By the way, not being able to do that seems to be one of the limitations of Reticulate, the R package for Python interop:
https://stackoverflow.com/questions/50762140/r-reticulate-how-can-i-close-restart-the-python-console
Hearing from people who use it intensively, that is indeed a practical problem. In cases of long running computations, as well as of memory leaks, being able to get rid of a python process and create a new one would be useful.)
When working with libpython-clj
from other projects or from journals like https://nextjournal.com/kommen/gigasquids-libpython-clj-examples, it's nice to be able to use a :git/url
and sha
to pin a specific commit - and not have to wait for a maven deployed version, (or have to clone the repo and install yourself.)
I'm happy to pitch in and look into converting to clojure deps and make it work with Circle CI if there is interest in this.
Currently, libpython-clj
offers syntax sugar for attribute lookup. However, the syntax is a bit inconsistent with Clojure's idioms.
(py/$. obj arg)
is an attribute lookup, whereas in Clojure (. obj method)
is a a method lookup.
Additionally, according to some anonymous unscientific sources, the $
symbol, originally invented by JavaScript, is a leading cause of carpal tunnel.
Therefore, the following syntax sugar is proposed:
(py. obj method arg1 arg2 ... argN)
-- method invocation
(py.- obj attr)
-- attribute lookup
(py.. obj -attr method-no-args (method with args))
-- nested access
Provide support so that users and library authors can define the behavior of datafy
and nav
for Python objects.
Currently the Python3 executable path is hardcoded to "python3"
. I propose we allow the executable to be specified via :python-executable
in py/initialize!
. On my system, Python3.6 always gets picked up first even though Python3.7 is available because Python3.6 is my current system alternative.
This change should also allow use of Python virtual environments.
Hello,
Thanks a lot for the library, it has been a real pleasure to use.
For now, we can call a function like this:
(py/$a np linspace :start 0 :stop_button: 1)
but it would be great if we could integrate a macro, say $a**
such that we could perform
(py/$a** np linspace {:start 0 :stop 1})
Obviously we could also do this
(defn $a** [x attr m] (py/call-attr-kw x attr nil m))
However, I think it would be a beneficial addition to the library.
Best regards,
David
Provide support for kebab case through the following require-python
flags:
(require-python '[libname :bind-case [:kebab :snake]])
(require-python '[libname :bind-case [:snake]])
(require-python '[libname :bind-case [:kebab]])
(require-python '[libname :kebab-case])
Ensure the following scenario is valid:
(ns a
(:require ...))
(require-python '[csv :bind-ns :kebab-case])
(defn something-with-csv [& args]
(csv/something-kebab args))
(ns b
(:require [a :refer [something-with-csv]] ...))
(require-python '[csv :bind-ns])
(defn something-else-with-csv [& args]
(-> args (csv/something_with_snake_case) (partial apply something-with-csv)))
Cases default to Python with opt-in for kebab.
This will allow the following API access:
(require-python '[csv :bind-ns :bind-case [:snake :kebab]])
(csv/field_size_limit)
(csv/field-size-limit)
(py. csv field_size_limit)
(py. csv field-size-limit)
(py.. csv field-size-limit)
(py.. csv field_size_limit)
user> (require-python '(mxnet mxnet.module))
nil
user> (require-python '[mxnet mxnet.module])
Execution error (IllegalArgumentException) at libpython-clj.require/python-lib-configuration (require.clj:306).
Vector arg to map conj must be a pair
I have tried running the example on osx.
The way to do that was:
sudo find / -name libpython3.6m.dylib
And taking the first results:
(alter-var-root #'libpython-clj.jna.base/python-library (constantly "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/config-3.6m-darwin/libpython3.6m.dylib"))
I was wondering if there is an easy way to get the current python based on the context (virtualenv, conda) and figure out the location of the libpython?
We need help with this. It is possible, it is a matter of figuring out the changes that conda makes to the system and what env variables the conda python shim sets up.
This is very important for newcomer experience so if you know anyone who knows how conda works or want to spend some time working with conda and libpython-clj please jump right in.
Currently there is a mismatch between require-python
syntax and require
syntax.
(require '(a b c))
implies (require 'a.b 'a.c)
.
Currently, require-python
syntax operates in the following manner:
(require-python '(a b c))
implies
(do
(require-python 'a)
(require-python 'b)
(require-python 'c))
This is incorrect.
(require-python '(a b c))
should also imply (require-python 'a.b 'a.c)
(which also doesn't work), and (require-python 'a.b 'a.c)
should imply
(do
(require-python 'a.b)
(require-python 'a.c))
Special attention should also be paid to allow for (require-python '(a [b :as c :refer [blah]]))
etc
which should imply (require-python '[a.b :as c :refer [blah]])
.
clj -A:new libpython-clj myapp/myname
would be pretty awesome!
But as a start,
clj -A:new https://github.com/cnuernber/libpython-clj@<sha> mydomain/myapp
would be good too!
How do I do Numpy style array indexing and slicing?
Numpy arrays (and PyTorch tensors) let you operate on them directly and efficiently.
As an example, how would I do something like this?
a[:, 5] += b[0] - a[:, 4].mean()
I don't want to do the indexing and slicing in the JVM. I want to do it in Python.
Attempted to use this with mxnet today. Here are thoughts so far:
It worked very well and provided good documentation and discoverability. This is awesome and a solid pathway forward.
(require-python '[builtins :alpha-load-ns-classes :as python])
but I cannot (require-python '[builtins.list :as python.list])
. This isn't pure Clojure syntax but it may be the best way.For instance, in mxnet, modules is a dynamic module beneath mxnet and it has a Module class with important methods on it. I want to type (mxnet.modules.Module/forward item batch-data)
and have intellisense help me out. This last point I think is actually the most important. I think we should be able to 'require' python classes as long as they are static variables of the python module. Ditto submodules.
In order to achieve some of the above in a consistent manner, I think we update the system so that it attempts to require the root module and then walk down using getattr to the target module. We then create the namespace and go from there. We don't assume the input path to require is a module itself but we do assume we can access it at runtime statically via get-attr calls.
I think also :alpha-load-ns-classes should just be renamed :recurse and there should be a list of exclusions :recurse-exclude that filters particularly expensive or crashy things out of the recursive pathway.
With these 2 changes and some solid testing I think this moves us another measurable step forward in terms of pure language integration.
Working hypothesis:
There is a nuanced difference between the behavior of
(f *args **kwargs)
and (py/call-kw f args kwargs)
in some rare edge cases that seem to come up frequently in Pandas.
It seems that all the breaking pathways lead to (f *args **kwargs)
and all the working pathways converge on (py/call-kw f args kwargs)
Each of the following examples converge to py/call-kw
.
(let [data (doto (pd/DataFrame {:index [1 2] :value [2 3] :variable [1 1]})
(py. melt :id_vars "index"))]
(py. px line :data_frame data :x "index" :y "value" :color "variable")) ;; works
(let [data (doto (pd/DataFrame {:index [1 2] :value [2 3] :variable [1 1]})
(py. melt :id_vars "index"))]
(py/call-attr-kw px "line" nil {:data_frame data :x "index" :y "value" :color "variable"})) ;; works
(let [data
(doto
(pd/DataFrame {:index [1 2] :value [2 3] :variable [1 1]})
(py. melt :id_vars "index"))]
(py/call-kw px/line nil {:data_frame data :x "index" :y "value" :color "variable"})) ;; works
(let [data
(doto
(pd/DataFrame {:index [1 2] :value [2 3] :variable [1 1]})
(py. melt :id_vars "index"))]
(py/call-kw (py.- px line ) nil {:data_frame data :x "index" :y "value" :color "variable"})) ;; works
(def px-line
(let [{{:keys [line]} :globals}
(py/run-simple-string
"
import plotly.express as px
line = px.line
")] line))
(let [data
(doto
(pd/DataFrame {:index [1 2] :value [2 3] :variable [1 1]})
(py. melt :id_vars "index"))]
(py/call-kw px-line nil {:data_frame data :x "index" :y "value" :color "variable"})) ;; works
Each of the following converge to (f *args **kwargs)
(let [data (doto (pd/DataFrame {:index [1 2] :value [2 3] :variable [1 1]})
(py. melt :id_vars "index"))]
((py.- px line) :data_frame data :x "index" :y "value" :color "variable")) ;; breaks
(let [data (doto (pd/DataFrame {:index [1 2] :value [2 3] :variable [1 1]})
(py. melt :id_vars "index"))]
(px/line :data_frame data :x "index" :y "value" :color "variable")) ;; breaks
(def px-line
(let [{{:keys [line]} :globals}
(py/run-simple-string
"
import plotly.express as px
line = px.line
")] line))
(let [data
(doto
(pd/DataFrame {:index [1 2] :value [2 3] :variable [1 1]})
(py. melt :id_vars "index"))]
(px-line :data_frame data :x "index" :y "value" :color "variable")) ;; breaks
(require '[clojure.core.async :reload true])
(require-python '[math :reload])
We should change the syntax to be consistent (and do it before the syntax percolates into code too far)
Hi,
I have a question regarding python array traversing.
What is the fastest way to traverse python array? The fastest variant I came up with:
(arr/array
array-type
(with-gil
(->> (range (libpy/PySequence_Length samples))
(mapv (fn [idx]
(let [time (/ idx frame-rate)
gain (* scale
(Math/sin (* twoPiF time)))]
(int (* gain
(-> (libpy/PySequence_GetItem samples idx)
wrap-pyobject
->jvm)))))))))
It is inefficient as I initialize a vector from input python array and after that converting that vector back to the python array.
According to benchmark, converting back to python array takes equal time that is needed to initialize vector from input.
So maybe there is some kind of better way iterating over python array and simultaneously
traversing its elements, so I can get a traversed version of an array as output, but unfortunately I couldn't figure out how to to do that.
Could you please help me to come up with better solution?
Thank you in advance for your effort and time
matplotlib-clj.core=> (plot-it)
2019-11-15 21:02:28.865 java[21046:1140194] WARNING: NSWindow drag regions should only be invalidated on the Main Thread! This will throw an exception in the future. Called from (
0 AppKit 0x00007fff2c69e6ef -[NSWindow(NSWindow_Theme) _postWindowNeedsToResetDragMarginsUnlessPostingDisabled] + 371
1 AppKit 0x00007fff2c69bbc1 -[NSWindow _initContent:styleMask:backing:defer:contentView:] + 1416
2 AppKit 0x00007fff2c69b633 -[NSWindow initWithContentRect:styleMask:backing:defer:] + 42
3 _macosx.cpython-37m-darwin.so 0x00000001361646b0 -[Window initWithContentRect:styleMask:backing:defer:withManager:] + 80
4 _macosx.cpython-37m-darwin.so 0x0000000136167bc7 FigureManager_init + 327
5 Python 0x000000012ddba050 wrap_init + 12
6 Python 0x000000012dd81407 wrapperdescr_call + 337
7 Python 0x000000012dd7b668 _PyObject_FastCallKeywords + 365
8 Python 0x000000012de123ad call_function + 746
9 Python 0x000000012de0af6f _PyEval_EvalFrameDefault + 7020
10 Python 0x000000012dd7bbdd function_code_fastcall + 112
11 Python 0x000000012dd7c561 _PyObject_Call_Prepend + 150
12 Python 0x000000012ddb9fbe slot_tp_init + 80
13 Python 0x000000012ddb6c85 type_call + 178
14 Python 0x000000012dd7b668 _PyObject_FastCallKeywords + 365
15 Python 0x000000012de123ad call_function + 746
16 Python 0x000000012de0af6f _PyEval_EvalFrameDefault + 7020
17 Python 0x000000012dd7bbdd function_code_fastcall + 112
18 Python 0x000000012de123b4 call_function + 753
19 Python 0x000000012de0af6f _PyEval_EvalFrameDefault + 7020
20 Python 0x000000012de12c5c _PyEval_EvalCodeWithName + 1857
21 Python 0x000000012dd7b42a _PyFunction_FastCallDict + 441
22 Python 0x000000012dd7c561 _PyObject_Call_Prepend + 150
23 Python 0x000000012dd7b910 PyObject_Call + 136
24 Python 0x000000012de0b286 _PyEval_EvalFrameDefault + 7811
25 Python 0x000000012de12c5c _PyEval_EvalCodeWithName + 1857
26 Python 0x000000012dd7b7c7 _PyFunction_FastCallKeywords + 225
27 Python 0x000000012de123b4 call_function + 753
28 Python 0x000000012de0b00d _PyEval_EvalFrameDefault + 7178
29 Python 0x000000012dd7bbdd function_code_fastcall + 112
30 Python 0x000000012de123b4 call_function + 753
31 Python 0x000000012de0b00d _PyEval_EvalFrameDefault + 7178
32 Python 0x000000012de12c5c _PyEval_EvalCodeWithName + 1857
33 Python 0x000000012dd7b7c7 _PyFunction_FastCallKeywords + 225
34 Python 0x000000012de123b4 call_function + 753
35 Python 0x000000012de0b00d _PyEval_EvalFrameDefault + 7178
36 Python 0x000000012de12c5c _PyEval_EvalCodeWithName + 1857
37 Python 0x000000012dd7b42a _PyFunction_FastCallDict + 441
38 jna2771800009792166730.tmp 0x0000000127a9be74 ffi_call_unix64 + 76
39 ??? 0x000070000ded2548 0x0 + 123145535956296
)
Reported here: https://clojurians.zulipchat.com/#narrow/stream/215609-libpython-clj-dev/topic/pandas.20query.20methods.20bug/near/186054336
(require-python
'([pandas :as pd]
[numpy :as np]))
pd ;; outputs the python outputs in 1.30
np ;; Error -> Unable to resolve symbol: os in this context in 1.31
#28 introduced two unit tests that can fail depending on how the test runner is invoked.
The tests in question occur:
The current-ns
might change depending on how the test runner is used.
A friend who uses python regularly indicated to me
"When it comes to python, sandboxed dependency environments are a must."
To that end I am able to do something like:
brew install pipenv
pipenv install seaborn
pipenv shell
python
then run the examples here: https://seaborn.pydata.org/introduction.html
However, the clojure seaborn
example will error out for me at this line: https://github.com/gigasquid/libpython-clj-examples/blob/master/src/gigasquid/seaborn.clj#L17
with
gigasquid.seaborn=> gigasquid.seaborn=> (require-python '[seaborn :as sns])
:ok
gigasquid.seaborn=> (require-python '[matplotlib.pyplot :as pyplot])
:ok
gigasquid.seaborn=> (sns/set)
nil
gigasquid.seaborn=> (def dots (sns/load_dataset "dots"))
Execution error at libpython-clj.python.interpreter/check-error-throw (interpreter.clj:474).
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 1317, in do_open
encode_chunked=req.has_header('Transfer-encoding'))
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 1229, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 1275, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 1224, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 1016, in _send_output
self.send(msg)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 956, in send
self.connect()
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 1392, in connect
server_hostname=server_hostname)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/ssl.py", line 412, in wrap_socket
session=session
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/ssl.py", line 850, in _create
self.do_handshake()
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/ssl.py", line 1108, in do_handshake
self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1045)
Is there a way to make it use the correct python, i.e. the one in the pipenv shell
?
Many thanks
Commit edd221d appears to have removed the def
for bridged
:
(def bridged (->> *1
(map (fn [[k v]]
[k (as-jvm v)]))
(into {})))
That makes the current code in the README:
(instance? java.util.Map (:globals bridged))
not work.
https://clojure.org/reference/reader#_symbols says:
Symbols beginning or ending with '.' are reserved by Clojure
In particular clojure uses them as syntax sugar for (new Whatever ...)
, this library defines several macros with names that end in '.' that for now happen to work as clojure itself doesn't enforce this reservation, but if it ever does it will break this library. And other tooling that respects this reservation breaks with this library
Thank you so much for this great project and I'm really excited of the further results :)
When I'm trying to test keras-simple
project on my virtualenv activated in the directory, it cannot import keras
module installed in the virtualenv but my REPL is unexpectedly killed instead.
I think this is really important support since many of python developers prefer to work in virtualenv. Is there any way to do it or do you have a plan for this functionality?
Thanks again for your work.
Thanks for this project! I enjoyed the talk from this year's Conj and it made me want to dive right in.
My use case is somewhat unique, I think: I'm trying to use Python to drive some hardware I have soldered to the GPIO pins on a Raspberry Pi Zero. The Pi is a 32-bit ARM processor, and Python works great on it... but Python interop has been giving me a bit of trouble.
Here's what we're working with:
$ ldconfig -p | grep python
libpython3.7m.so.1.0 (libc6,hard-float) => /usr/lib/arm-linux-gnueabihf/libpython3.7m.so.1.0
libpython3.7m.so (libc6,hard-float) => /usr/lib/arm-linux-gnueabihf/libpython3.7m.so
libpython2.7.so.1.0 (libc6,hard-float) => /usr/lib/arm-linux-gnueabihf/libpython2.7.so.1.0
libpython2.7.so (libc6,hard-float) => /usr/lib/arm-linux-gnueabihf/libpython2.7.so
and Java:
$ java -version
openjdk version "1.8.0_152"
OpenJDK Runtime Environment (Zulu Embedded 8.25.0.76-linux-aarch32hf) (build 1.8.0_152-b76)
OpenJDK Client VM (Zulu Embedded 8.25.0.76-linux-aarch32hf) (build 25.152-b76, mixed mode, Evaluation)
I'm using Python 3.7 to drive the hardware, and Python itself runs fine.
Likewise, I can run Clojure (both JARs and Clojure in the REPL) without any issue.
But when I try to run Python from within Clojure via the REPL, here's what happens:
=> (require '[libpython-clj.python :as py])
nil
=> (py/initialize!)
Jan 09, 2020 3:05:16 AM clojure.tools.logging$eval1906$fn__1909 invoke
INFO: Executing python initialize!
Jan 09, 2020 3:05:17 AM clojure.tools.logging$eval1906$fn__1909 invoke
INFO: Startup info detected: {:python-home "/usr", :lib-version "3.7", :libname "python3.7m", :java-library-path-addendum "/usr/lib"}
Jan 09, 2020 3:05:17 AM clojure.tools.logging$eval1906$fn__1909 invoke
INFO: Reference thread starting
Jan 09, 2020 3:05:18 AM clojure.tools.logging$eval1906$fn__1909 invoke
INFO: Library python3.7m found at [:system "python3.7m"]
Execution error (IllegalArgumentException) at libpython-clj.jna.base/eval22533$fn$G (base.clj:20).
No implementation of method: :->py-object-ptr of protocol: #'libpython-clj.jna.base/PToPyObjectPtr found for class: nil
The error repeats itself if you try any other commands:
=> (py/run-simple-string "print('Hello')")
Execution error (IllegalArgumentException) at libpython-clj.jna.base/eval22533$fn$G (base.clj:20).
No implementation of method: :->py-object-ptr of protocol: #'libpython-clj.jna.base/PToPyObjectPtr found for class: nil
I started the repl using lein repl
in a new project root.
The project was scaffolded with the lein new app
command.
Do you see anything off the top of your head that may be the root cause of these issues?
My gut tells me it's one of three things:
Any thoughts as to why this is happening, or is there any more information I could provide?
Really appreciate your time. Thank you!
This avoids issues like:
lein test :only panthera.generics-test/head
ERROR in (head) (interpreter.clj:385)
expected: (= (u/->clj (g/head (g/data-frame (flatten (repeat 5 [{:a 1, :b 2} {:a 2, :
actual: java.lang.Exception: Traceback (most recent call last):
File "/mnt/DATA/miniconda3/envs/panthera/lib/python3.7/site-packages/pandas/core/fr
elif isinstance(data, abc.Iterable) and not isinstance(data, (str, bytes)):
File "/mnt/DATA/miniconda3/envs/panthera/lib/python3.7/abc.py", line 139, in __inst
return _abc_instancecheck(cls, instance)
File "/mnt/DATA/miniconda3/envs/panthera/lib/python3.7/abc.py", line 143, in __subc
return _abc_subclasscheck(cls, subclass)
TypeError: issubclass() arg 1 must be a class
From the slack channel
is it possible in the clojure machinery to do :refer-clojure :exclude :all: in the generated namespaces
the warnings about "java.lang.Exception is being overridden" are a bit much
Hello, I am learning about ML and trying to port https://github.com/pytorch/examples/blob/master/mnist/main.py
to libpython-clj since I love Clojure.
Quickly scanning through the code and documentation, I do not see any way to subclass the torch.nn.Module
into your own neural net from the Clojure side. Am I missing something, or is there an example somewhere?
Hi @cnuernber,
thank you for the great library. I just wonder whether is it possible to utilize type checking for Python types. Unfortunately, I haven't found any reference in a source code.
Thanks in advance
Hello,
First of all, this is a great project and I wish we could see it being the standard way to interact with python and Clojure!
It is only a small detail with respect to the all difficulties you might have for creating a bridge between python and Clojure, but would it make sense to create some sort of sugar/interface/macro to call functions from modules with would look like either namespace calling, e.g np/zeros
for modules or like (.attr object & args)
or
(.attr object & kwargs)
for modules and object.
libpython-clj seems hang nrepl if I have multiple nrepl client sessions connected to the same nrepl server (typically one from terminal, one from cursive/calva ide).
Here are some stacktraces, which show interpreter/with-gil could cause some kind of deadlock. e.g. one thread (#54) locked on java lock <0x000000071596e020> then block on native jna call indefinitely. All other nrepl threads that uses interpreter/with-gil hang on java.lang.Thread.State: BLOCKED (on object monitor) - waiting to lock <0x000000071596e020>
None of these threads made any process when I did multiple thread dumps. Hence, I suspect it ended up in a complete deadlock condition.
"nRepl-session-c7872ab8-6dca-4432-a4c1-04712fec8945" #54 daemon prio=5 os_prio=0 cpu=1315.60ms elapsed=1109.28s tid=0x00007f947c268000 nid=0x30a runnable [0x00007f94deef5000]
java.lang.Thread.State: RUNNABLE
at com.sun.jna.Native.invokePointer(Native Method)
at com.sun.jna.Function.invokePointer(Function.java:496)
at com.sun.jna.Function.invoke(Function.java:440)
at com.sun.jna.Function.invoke(Function.java:360)
at com.sun.jna.Function.invoke(Function.java:314)
at com.sun.jna.Function.invoke(Function.java:305)
at libpython_clj.jna.protocols.object$PyObject_Call.invokeStatic(object.clj:277)
at libpython_clj.jna.protocols.object$PyObject_Call.invoke(object.clj:277)
at clojure.lang.Var.invoke(Var.java:393)
at libpython_clj.python.object$eval41499$fn__41500$fn__41501.invoke(object.clj:828)
at libpython_clj.python.interpreter$with_gil_fn.invokeStatic(interpreter.clj:377)
at libpython_clj.python.interpreter$with_gil_fn.invoke(interpreter.clj:346)
at libpython_clj.python.object$eval41499$fn__41500.invoke(object.clj:825)
at libpython_clj.python.protocols$eval39373$fn__39374$G__39364__39383.invoke(protocols.clj:105)
at libpython_clj.python.bridge$generic_python_as_jvm$fn$reify__42101$fn__42138.invoke(bridge.clj:633)
at libpython_clj.python.interpreter$with_gil_fn.invokeStatic(interpreter.clj:377)
at libpython_clj.python.interpreter$with_gil_fn.invoke(interpreter.clj:346)
at libpython_clj.python.bridge$generic_python_as_jvm$fn$reify__42101.do_call_fn(bridge.clj:633)
at libpython_clj.python.protocols$call_kw.invokeStatic(protocols.clj:118)
at libpython_clj.python.protocols$call_kw.invoke(protocols.clj:115)
at libpython_clj.python.bridge$cfn.invokeStatic(bridge.clj:595)
at libpython_clj.python.bridge$cfn.doInvoke(bridge.clj:586)
at clojure.lang.RestFn.invoke(RestFn.java:460)
at libpython_clj.python.bridge$generic_python_as_jvm$fn$reify__42101$fn__42130.invoke(bridge.clj:661)
at libpython_clj.python.interpreter$with_gil_fn$fn__38951$fn__38952.invoke(interpreter.clj:360)
at clojure.lang.AFn.applyToHelper(AFn.java:152)
at clojure.lang.AFn.applyTo(AFn.java:144)
at clojure.core$apply.invokeStatic(core.clj:665)
at clojure.core$with_bindings_STAR_.invokeStatic(core.clj:1973)
at clojure.core$with_bindings_STAR_.doInvoke(core.clj:1973)
at clojure.lang.RestFn.invoke(RestFn.java:425)
at libpython_clj.python.interpreter$with_gil_fn$fn__38951.invoke(interpreter.clj:358)
at clojure.lang.AFn.applyToHelper(AFn.java:152)
at clojure.lang.AFn.applyTo(AFn.java:144)
at clojure.core$apply.invokeStatic(core.clj:665)
at clojure.core$with_bindings_STAR_.invokeStatic(core.clj:1973)
at clojure.core$with_bindings_STAR_.doInvoke(core.clj:1973)
at clojure.lang.RestFn.invoke(RestFn.java:425)
at libpython_clj.python.interpreter$with_gil_fn.invokeStatic(interpreter.clj:356)
- locked **<0x000000071596e020>** (a libpython_clj.python.interpreter.Interpreter)
at libpython_clj.python.interpreter$with_gil_fn.invoke(interpreter.clj:346)
at libpython_clj.python.bridge$generic_python_as_jvm$fn$reify__42101.invoke(bridge.clj:660)
"nRepl-session-45ddd45e-a1f6-427e-ba96-e37f54064267" #57 daemon prio=5 os_prio=0 cpu=1.86ms elapsed=403.02s tid=0x00007f96b0007000 nid=0x9a7 waiting for monitor entry [0x00007f94df2fa000]
java.lang.Thread.State: BLOCKED (on object monitor)
at libpython_clj.python.interpreter$with_gil_fn.invokeStatic(interpreter.clj:354)
- waiting to lock **<0x000000071596e020>** (a libpython_clj.python.interpreter.Interpreter)
at libpython_clj.python.interpreter$with_gil_fn.invoke(interpreter.clj:346)
at libpython_clj.python.bridge$generic_python_as_jvm$fn$reify__42153.get_attr(bridge.clj:684)
at libpython_clj.python.protocols$call_attr_kw.invokeStatic(protocols.clj:131)
at libpython_clj.python.protocols$call_attr_kw.invoke(protocols.clj:127)
at clojure.lang.Var.invoke(Var.java:399)
at gpt2$generate_text.invokeStatic(gpt2.clj:44)
at gpt2$generate_text.invoke(gpt2.clj:43)
at gpt2$eval46978.invokeStatic(form-init3818583985170112959.clj:1)
at gpt2$eval46978.invoke(form-init3818583985170112959.clj:1)
at clojure.lang.Compiler.eval(Compiler.java:7177)
at clojure.lang.Compiler.eval(Compiler.java:7132)
at clojure.core$eval.invokeStatic(core.clj:3214)
at clojure.core$eval.invoke(core.clj:3210)
at clojure.main$repl$read_eval_print__9086$fn__9089.invoke(main.clj:437)
at clojure.main$repl$read_eval_print__9086.invoke(main.clj:437)
at clojure.main$repl$fn__9095.invoke(main.clj:458)
at clojure.main$repl.invokeStatic(main.clj:458)
at clojure.main$repl.doInvoke(main.clj:368)
at clojure.lang.RestFn.invoke(RestFn.java:1523)
at nrepl.middleware.interruptible_eval$evaluate.invokeStatic(interruptible_eval.clj:79)
at nrepl.middleware.interruptible_eval$evaluate.invoke(interruptible_eval.clj:55)
at nrepl.middleware.interruptible_eval$interruptible_eval$fn__946$fn__950.invoke(interruptible_eval.clj:142)
at clojure.lang.AFn.run(AFn.java:22)
at nrepl.middleware.session$session_exec$main_loop__1047$fn__1051.invoke(session.clj:171)
at nrepl.middleware.session$session_exec$main_loop__1047.invoke(session.clj:170)
at clojure.lang.AFn.run(AFn.java:22)
at java.lang.Thread.run([email protected]/Thread.java:834)
Uberjar's using require-python are currently failing.
We at least need to dig in, investigate, and document how to work around this if not fix it.
Hi!
Thank you so much for this awesome library. We are currently experimenting with it and it seems to deliver big advantages for us compared to RPC through TCP/Protobuf that we have used before communicating between python and clojure-processes.
I hit a very odd issue on MacOS X, with Python 3.7.6_1 with libpython python3.7m.
If I require the sys
module as the first requirement, it fails with a message:
maharj:javascript $ clj -Sdeps '{:deps {clj-python/libpython-clj {:mvn/version "1.36"}}}'
Clojure 1.10.1
user=> (require '[libpython-clj.require :as pyr])
Feb 20, 2020 8:50:41 AM clojure.tools.logging$eval496$fn__499 invoke
INFO: Executing python initialize with options:{:python-executable nil, :program-name nil, :python-home nil, :library-path nil}
Feb 20, 2020 8:50:41 AM clojure.tools.logging$eval496$fn__499 invoke
INFO: Detecting startup-info for Python executable:
Feb 20, 2020 8:50:41 AM clojure.tools.logging$eval496$fn__499 invoke
INFO: Startup info detected: {:python-home "/usr/local/Cellar/python/3.7.6_1/Frameworks/Python.framework/Versions/3.7", :lib-version "3.7", :libname "python3.7m", :java-library-path-addendum "/usr/local/Cellar/python/3.7.6_1/Frameworks/Python.framework/Versions/3.7/lib"}
Feb 20, 2020 8:50:41 AM clojure.tools.logging$eval496$fn__499 invoke
INFO: Setting java library path: /usr/local/Cellar/python/3.7.6_1/Frameworks/Python.framework/Versions/3.7/lib:/Users/maharj/Library/Java/Extensions:/Library/Java/Extensions:/Network/Library/Java/Extensions:/System/Library/Java/Extensions:/usr/lib/java:.
Feb 20, 2020 8:50:41 AM clojure.tools.logging$eval496$fn__499 invoke
INFO: Reference thread starting
Feb 20, 2020 8:50:41 AM clojure.tools.logging$eval496$fn__499 invoke
INFO: Library python3.7m found at [:java-library-path "/usr/local/Cellar/python/3.7.6_1/Frameworks/Python.framework/Versions/3.7/lib/libpython3.7m.dylib"]
Feb 20, 2020 8:50:41 AM clojure.tools.logging$eval496$fn__499 invoke
INFO: Library c found at [:system "c"]
nil
user=> (pyr/require-python 'sys)
Execution error (ExceptionInfo) at libpython-clj.jna.base/ensure-pyobj (base.clj:52).
Failed to get a pyobject pointer from object.
user=> (pyr/require-python 'sys)
:ok
However, if I import another module, say os
before, it works like expected.
I did not have time to inspect this more at this time, but as I've stepped through the code with debugger, the problem disappears on some occasion, so maybe this is due to some timing issue? It does not, however correllate with the time one waits between executing the (pyr/require 'sys)
after requiring the library (as it does its initialization on the start of the load, one could expect that it would have some effect...).
Thanks!
expected:
(python/str \a) ;;=> "a"
actual: 💥
Unable to convert objects of type: class java.lang.Character
Hello Chris!
I really enjoyed listening to Scicloj meeting #4. You're doing interesting work here. I feel that you're setting the Clojure community up for an interop story with Python that's similar to how we're able to make use of Java.
I have a question, if you don't mind. Per now, I can use my own interpreter of choice like this:
(require '[libpython-clj.python :as py]
'[libpython-clj.jna.base])
(defn get-python [_]
"/home/teodorlu/opt/anaconda3/bin/python")
(alter-var-root #'libpython-clj.jna.base/*python-library* #'get-python)
I would like to use libpython-clj
to interface with a running Python process that's not started from within the Clojure process, but "on its own terms". A possible process could look like this:
libpython-clj
to ... do what it needs to do.The reason why I can't just start a new process, is that the Python process is started by another application, and I need access to the application state, as the user interacts with the state.
libpython-clj
?I spent some time reading the libpython-clj
source code, but I must admit that this is above my level, hence I'm asking.
Thanks again for your work, and have a nice day!
Teodor
Once again, amazing project!
I was experimenting if it's possible to use flax with libpython-clj. I'm pretty much asking for trouble by combining these early-stage libraries, but it's just for fun. It seems that the code below fails because the function apply
that libpython-clj defines on the fly does not have a Function Signature Object associated with it, which flax
tries to use via inspect.signature
. Or I'm missing something in my Clojure code?
Just wondering if this is a known limitation, or there is a way to include this signature? Feel free to close the issue if this is not a top priority.
(ns flax_model
(:require [libpython-clj.require :refer [require-python]]
[libpython-clj.python :as py]]))
(require-python '[flax.nn :as nn])
(def identity (py/create-class "Identity" [nn/Module]
{"apply" (py/make-tuple-instance-fn
(fn [self x] x)
:arg-converter py/as-jvm
:method-name "apply")}))
Stack trace:
Execution error at libpython-clj.python.interpreter/check-error-throw (interpreter.clj:474).
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/flax/nn/base.py", line 169, in __init__
apply_params = _fn_parameters(apply_fn)
File "/usr/local/lib/python3.7/site-packages/flax/nn/base.py", line 153, in _fn_parameters
return tuple(inspect.signature(fn).parameters.values())
File "/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/lib/python3.7/inspect.py", line 3083, in signature
return Signature.from_callable(obj, follow_wrapped=follow_wrapped)
File "/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/lib/python3.7/inspect.py", line 2833, in from_callable
follow_wrapper_chains=follow_wrapped)
File "/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/lib/python3.7/inspect.py", line 2288, in _signature_from_callable
skip_bound_arg=skip_bound_arg)
File "/usr/local/Cellar/python/3.7.4/Frameworks/Py
thon.framework/Versions/3.7/lib/python3.7/inspect.py", line 2112, in _signature_from_builtin
raise ValueError("no signature found for builtin {!r}".format(func))
ValueError: no signature found for builtin <built-in function apply>
class clojure.lang.Compiler$CompilerException
Not all python libraries tolerate multithreaded access well, even when serialized by a GIL. Some, for instance, use static/thread-local variables in a way as to make then unsafe to use from multiple threads.
We want to make purely single threaded access possible when necessary for robustness. It should be possible to load python and use it from an agent and guarantee all python access is from one thread.
One potential soution is a custom gc release-refcount queue that is cleared at the beginning or end of with-gil.
First of all, THANK YOU for building this. It's exactly what I've wanted for so long!
I'm trying to use ta-lib in my REPL to pull in various technical indicators and use matplotlib to chart the series. It works 90% of the time, but every tenth time or so, my application crashes with the below stack trace. Any idea what I should be avoiding or what can cause this? Many thanks!
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGSEGV (0xb) at pc=0x00007f5a1f876211, pid=635, tid=709
#
# JRE version: OpenJDK Runtime Environment (11.0.6+10) (build 11.0.6+10)
# Java VM: OpenJDK 64-Bit Server VM (11.0.6+10, mixed mode, tiered, compressed oops, g1 gc, linux-amd64)
# Problematic frame:
# C [libpython3.7m.so+0x276211] _PyObject_FastCallDict+0x71
#
# No core dump will be written. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# An error report file with more information is saved as:
# /app/hs_err_pid635.log
#
# If you would like to submit a bug report, please visit:
# http://bugreport.java.com/bugreport/crash.jsp
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
This is both a robustness upgrade and performance upgrade. The current interpreter design was made as a midpoint between an single and multiple interpreter design. We have discussed a multiple interpreter design at length and decided it wasn't necessary. The best pathway past the GIL really is a subprocess and not a thread or multiple interpreters.
So it is time to revisit the interpreter design accounting for the fact that there will only be one interpreter. This is a major simplification of the library just right off the bat.
Also, the use of thread local variables can be somewhat optimized. Clojure's dynamic variables are extremely convenient but a concurrent thread local hash map (or potentially just an atomic long object) to indicate which thread owns the interpreter is both simple, fast, and fine.
This can be done without changing the external and I think most of the internal apis. Potentially just python/interpreter.clj.
Expected:
(try
(throw python/Exception)
(catch python/Exception e
:win)) ;;=> :win
Actual: 💥
It would be great to have a convenient syntax to allow Python generators to be in a Clojure transducing context!
Currently load-python-lib
is very procedural and therefore difficult to test. I suggest refactoring to use a pattern of a series of functions which build up a map of information, followed by a series of procedures which take action on that map.
If the side-effecty procedures which take actions on the load-python-lib-map
are small and well understood, it will make debugging easier and confidence higher.
Similarly, the map produced by the series of data transformations in the preceding step will be able to be tested.
As discussed here.
We now have more or less generalized binding for python to datafy/nav. We think, more or less :-).
If someone is so inclined, some testing/tutorials around annotating some python library and seeing what happens when they try to datafy/nav it via REBL would be cool.
In Python, object.method(*args, **kwargs)
is equivalent to class.method(object, *args, **kwargs)
.
From a user perspective, the class-method-object syntax is usually indistinguishable from module-function syntax, aka, math.sin(1)
. A concrete example:
>>> l = [1, 2, 3]
>>> list.append(l, 4)
>>> l
[1, 2, 3, 4]
could just as easily be accomplished in libpython-clj via
(def l (python/list [1 2 3]))
(python.list/append 4)
l ;;=> [1, 2, 3, 4]
This may require special corresponding methods such as py/set-ns-class-attr!
if mutating class attributes are necessary. This may also require additionally bookkeeping in require-python
to make sure that when a namespace is reloaded, all corresponding ns-classes are also reloaded.
Sequences (1.24+) are breaking how panthera works, basically this works:
(pt/data-frame (vec (flatten (repeat 5 [{:a 1 :b 2} {:a 2 :b 3}]))))
But this doesn't:
(pt/data-frame (flatten (repeat 5 [{:a 1 :b 2} {:a 2 :b 3}])))
Execution error at libpython-clj.python.interpreter/check-error-throw (interpreter.clj:506).
Traceback (most recent call last):
File "/mnt/DATA/miniconda3/envs/panthera/lib/python3.7/site-packages/pandas/core/frame.py", line 443, in __init__
elif isinstance(data, abc.Iterable) and not isinstance(data, (str, bytes)):
File "/mnt/DATA/miniconda3/envs/panthera/lib/python3.7/abc.py", line 139, in __instancecheck__
return _abc_instancecheck(cls, instance)
File "/mnt/DATA/miniconda3/envs/panthera/lib/python3.7/abc.py", line 143, in __subclasscheck__
return _abc_subclasscheck(cls, subclass)
TypeError: issubclass() arg 1 must be a class
Just as a reference, this breaks in the same way:
(require-python '[pandas :as pd])
(pd/DataFrame (flatten (repeat 5 [{:a 1 :b 2} {:a 2 :b 3}])))
I'm already sanitizing user input a lot (https://github.com/alanmarazzi/panthera/blob/e16d254d7d8048af0381cbf2e3285a77db517fb2/src/panthera/pandas/utils.clj#L229) because pandas expects fully realized objects shaped in a certain way to actually work.
P.S.: refer to the dev branch since there's a lot of stuff that isn't in master yet
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.