clj-python / libpython-clj Goto Github PK

#28 introduced two unit tests that can fail depending on how the test runner is invoked.

The tests in question occur:

• https://github.com/cnuernber/libpython-clj/blob/master/test/libpython_clj/require_python_test.clj#L112
• https://github.com/cnuernber/libpython-clj/blob/master/test/libpython_clj/require_python_test.clj#L77

The current-ns might change depending on how the test runner is used.

libpython-clj seems hang nrepl if I have multiple nrepl client sessions connected to the same nrepl server (typically one from terminal, one from cursive/calva ide).

Here are some stacktraces, which show interpreter/with-gil could cause some kind of deadlock. e.g. one thread (#54) locked on java lock <0x000000071596e020> then block on native jna call indefinitely. All other nrepl threads that uses interpreter/with-gil hang on java.lang.Thread.State: BLOCKED (on object monitor) - waiting to lock <0x000000071596e020>

None of these threads made any process when I did multiple thread dumps. Hence, I suspect it ended up in a complete deadlock condition.

"nRepl-session-c7872ab8-6dca-4432-a4c1-04712fec8945" #54 daemon prio=5 os_prio=0 cpu=1315.60ms elapsed=1109.28s tid=0x00007f947c268000 nid=0x30a runnable  [0x00007f94deef5000]
   java.lang.Thread.State: RUNNABLE
        at com.sun.jna.Native.invokePointer(Native Method)
        at com.sun.jna.Function.invokePointer(Function.java:496)
        at com.sun.jna.Function.invoke(Function.java:440)
        at com.sun.jna.Function.invoke(Function.java:360)
        at com.sun.jna.Function.invoke(Function.java:314)
        at com.sun.jna.Function.invoke(Function.java:305)
        at libpython_clj.jna.protocols.object$PyObject_Call.invokeStatic(object.clj:277)
        at libpython_clj.jna.protocols.object$PyObject_Call.invoke(object.clj:277)
        at clojure.lang.Var.invoke(Var.java:393)
        at libpython_clj.python.object$eval41499$fn__41500$fn__41501.invoke(object.clj:828)
        at libpython_clj.python.interpreter$with_gil_fn.invokeStatic(interpreter.clj:377)
        at libpython_clj.python.interpreter$with_gil_fn.invoke(interpreter.clj:346)
        at libpython_clj.python.object$eval41499$fn__41500.invoke(object.clj:825)
        at libpython_clj.python.protocols$eval39373$fn__39374$G__39364__39383.invoke(protocols.clj:105)
        at libpython_clj.python.bridge$generic_python_as_jvm$fn$reify__42101$fn__42138.invoke(bridge.clj:633)
        at libpython_clj.python.interpreter$with_gil_fn.invokeStatic(interpreter.clj:377)
        at libpython_clj.python.interpreter$with_gil_fn.invoke(interpreter.clj:346)
        at libpython_clj.python.bridge$generic_python_as_jvm$fn$reify__42101.do_call_fn(bridge.clj:633)
        at libpython_clj.python.protocols$call_kw.invokeStatic(protocols.clj:118)
        at libpython_clj.python.protocols$call_kw.invoke(protocols.clj:115)
        at libpython_clj.python.bridge$cfn.invokeStatic(bridge.clj:595)
        at libpython_clj.python.bridge$cfn.doInvoke(bridge.clj:586)
        at clojure.lang.RestFn.invoke(RestFn.java:460)
        at libpython_clj.python.bridge$generic_python_as_jvm$fn$reify__42101$fn__42130.invoke(bridge.clj:661)
        at libpython_clj.python.interpreter$with_gil_fn$fn__38951$fn__38952.invoke(interpreter.clj:360)
        at clojure.lang.AFn.applyToHelper(AFn.java:152)
        at clojure.lang.AFn.applyTo(AFn.java:144)
        at clojure.core$apply.invokeStatic(core.clj:665)
        at clojure.core$with_bindings_STAR_.invokeStatic(core.clj:1973)
        at clojure.core$with_bindings_STAR_.doInvoke(core.clj:1973)
        at clojure.lang.RestFn.invoke(RestFn.java:425)
        at libpython_clj.python.interpreter$with_gil_fn$fn__38951.invoke(interpreter.clj:358)
        at clojure.lang.AFn.applyToHelper(AFn.java:152)
        at clojure.lang.AFn.applyTo(AFn.java:144)
        at clojure.core$apply.invokeStatic(core.clj:665)
        at clojure.core$with_bindings_STAR_.invokeStatic(core.clj:1973)
        at clojure.core$with_bindings_STAR_.doInvoke(core.clj:1973)
        at clojure.lang.RestFn.invoke(RestFn.java:425)
        at libpython_clj.python.interpreter$with_gil_fn.invokeStatic(interpreter.clj:356)
        - locked **<0x000000071596e020>** (a libpython_clj.python.interpreter.Interpreter)
        at libpython_clj.python.interpreter$with_gil_fn.invoke(interpreter.clj:346)
        at libpython_clj.python.bridge$generic_python_as_jvm$fn$reify__42101.invoke(bridge.clj:660)

"nRepl-session-45ddd45e-a1f6-427e-ba96-e37f54064267" #57 daemon prio=5 os_prio=0 cpu=1.86ms elapsed=403.02s tid=0x00007f96b0007000 nid=0x9a7 waiting for monitor entry  [0x00007f94df2fa000]
   java.lang.Thread.State: BLOCKED (on object monitor)
        at libpython_clj.python.interpreter$with_gil_fn.invokeStatic(interpreter.clj:354)
        - waiting to lock **<0x000000071596e020>** (a libpython_clj.python.interpreter.Interpreter)
        at libpython_clj.python.interpreter$with_gil_fn.invoke(interpreter.clj:346)
        at libpython_clj.python.bridge$generic_python_as_jvm$fn$reify__42153.get_attr(bridge.clj:684)
        at libpython_clj.python.protocols$call_attr_kw.invokeStatic(protocols.clj:131)
        at libpython_clj.python.protocols$call_attr_kw.invoke(protocols.clj:127)
        at clojure.lang.Var.invoke(Var.java:399)
        at gpt2$generate_text.invokeStatic(gpt2.clj:44)
        at gpt2$generate_text.invoke(gpt2.clj:43)
        at gpt2$eval46978.invokeStatic(form-init3818583985170112959.clj:1)
        at gpt2$eval46978.invoke(form-init3818583985170112959.clj:1)
        at clojure.lang.Compiler.eval(Compiler.java:7177)
        at clojure.lang.Compiler.eval(Compiler.java:7132)
        at clojure.core$eval.invokeStatic(core.clj:3214)
        at clojure.core$eval.invoke(core.clj:3210)
        at clojure.main$repl$read_eval_print__9086$fn__9089.invoke(main.clj:437)
        at clojure.main$repl$read_eval_print__9086.invoke(main.clj:437)
        at clojure.main$repl$fn__9095.invoke(main.clj:458)
        at clojure.main$repl.invokeStatic(main.clj:458)
        at clojure.main$repl.doInvoke(main.clj:368)
        at clojure.lang.RestFn.invoke(RestFn.java:1523)
        at nrepl.middleware.interruptible_eval$evaluate.invokeStatic(interruptible_eval.clj:79)
        at nrepl.middleware.interruptible_eval$evaluate.invoke(interruptible_eval.clj:55)
        at nrepl.middleware.interruptible_eval$interruptible_eval$fn__946$fn__950.invoke(interruptible_eval.clj:142)
        at clojure.lang.AFn.run(AFn.java:22)
        at nrepl.middleware.session$session_exec$main_loop__1047$fn__1051.invoke(session.clj:171)
        at nrepl.middleware.session$session_exec$main_loop__1047.invoke(session.clj:170)
        at clojure.lang.AFn.run(AFn.java:22)
        at java.lang.Thread.run([email protected]/Thread.java:834)

Working hypothesis:

There is a nuanced difference between the behavior of
(f *args **kwargs) and (py/call-kw f args kwargs) in some rare edge cases that seem to come up frequently in Pandas.

It seems that all the breaking pathways lead to (f *args **kwargs) and all the working pathways converge on (py/call-kw f args kwargs)

Working

Each of the following examples converge to py/call-kw.

(let [data (doto (pd/DataFrame {:index [1 2] :value [2 3] :variable [1 1]})
               (py. melt :id_vars "index"))]
    (py. px line :data_frame data :x "index" :y "value" :color "variable")) ;; works

(let [data (doto (pd/DataFrame {:index [1 2] :value [2 3] :variable [1 1]})
               (py. melt :id_vars "index"))]
    (py/call-attr-kw px "line" nil {:data_frame data :x "index" :y "value" :color "variable"})) ;; works

(let [data
        (doto
            (pd/DataFrame {:index [1 2] :value [2 3] :variable [1 1]})
          (py. melt :id_vars "index"))]
    (py/call-kw px/line nil {:data_frame data :x "index" :y "value" :color "variable"})) ;; works

(let [data
        (doto
            (pd/DataFrame {:index [1 2] :value [2 3] :variable [1 1]})
          (py. melt :id_vars "index"))]
    (py/call-kw (py.- px line ) nil {:data_frame data :x "index" :y "value" :color "variable"})) ;; works

(def px-line
    (let [{{:keys [line]} :globals}
          (py/run-simple-string
           "
import plotly.express as px

line = px.line
")] line))

(let [data
        (doto
            (pd/DataFrame {:index [1 2] :value [2 3] :variable [1 1]})
          (py. melt :id_vars "index"))]
    (py/call-kw px-line  nil {:data_frame data :x "index" :y "value" :color "variable"})) ;; works

Breaking

Each of the following converge to (f *args **kwargs)

(let [data (doto (pd/DataFrame {:index [1 2] :value [2 3] :variable [1 1]})
               (py. melt :id_vars "index"))]
    ((py.- px line) :data_frame data :x "index" :y "value" :color "variable")) ;; breaks

(let [data (doto (pd/DataFrame {:index [1 2] :value [2 3] :variable [1 1]})
               (py. melt :id_vars "index"))]
    (px/line :data_frame data :x "index" :y "value" :color "variable")) ;; breaks

  (def px-line
    (let [{{:keys [line]} :globals}
          (py/run-simple-string
           "
import plotly.express as px

line = px.line
")] line))
  
  (let [data
        (doto
            (pd/DataFrame {:index [1 2] :value [2 3] :variable [1 1]})
          (py. melt :id_vars "index"))]
    (px-line :data_frame data :x "index" :y "value" :color "variable")) ;; breaks

Provide support for kebab case through the following require-python flags:

(require-python '[libname :bind-case [:kebab :snake]])
(require-python '[libname :bind-case [:snake]])
(require-python '[libname :bind-case [:kebab]])
(require-python '[libname :kebab-case])

Ensure the following scenario is valid:

(ns a
  (:require ...))

(require-python '[csv :bind-ns :kebab-case])

(defn something-with-csv [& args]
  (csv/something-kebab args))

(ns b
  (:require [a :refer [something-with-csv]]  ...))

(require-python '[csv :bind-ns])

(defn something-else-with-csv [& args]
  (-> args (csv/something_with_snake_case) (partial apply something-with-csv)))

Cases default to Python with opt-in for kebab.

This will allow the following API access:

(require-python '[csv :bind-ns :bind-case [:snake :kebab]])

(csv/field_size_limit)
(csv/field-size-limit)
(py. csv field_size_limit)
(py. csv field-size-limit)
(py.. csv field-size-limit)
(py.. csv field_size_limit)

This is both a robustness upgrade and performance upgrade. The current interpreter design was made as a midpoint between an single and multiple interpreter design. We have discussed a multiple interpreter design at length and decided it wasn't necessary. The best pathway past the GIL really is a subprocess and not a thread or multiple interpreters.

So it is time to revisit the interpreter design accounting for the fact that there will only be one interpreter. This is a major simplification of the library just right off the bat.

Also, the use of thread local variables can be somewhat optimized. Clojure's dynamic variables are extremely convenient but a concurrent thread local hash map (or potentially just an atomic long object) to indicate which thread owns the interpreter is both simple, fast, and fine.

This can be done without changing the external and I think most of the internal apis. Potentially just python/interpreter.clj.

Currently, libpython-clj offers syntax sugar for attribute lookup. However, the syntax is a bit inconsistent with Clojure's idioms.

(py/$. obj arg) is an attribute lookup, whereas in Clojure (. obj method) is a a method lookup.

Additionally, according to some anonymous unscientific sources, the $ symbol, originally invented by JavaScript, is a leading cause of carpal tunnel.

Therefore, the following syntax sugar is proposed:

(py. obj method arg1 arg2 ... argN) -- method invocation
(py.- obj attr) -- attribute lookup
(py.. obj -attr method-no-args (method with args)) -- nested access

It would be great to have a convenient syntax to allow Python generators to be in a Clojure transducing context!

How do I do Numpy style array indexing and slicing?

Numpy arrays (and PyTorch tensors) let you operate on them directly and efficiently.

As an example, how would I do something like this?

a[:, 5] += b[0] - a[:, 4].mean()

I don't want to do the indexing and slicing in the JVM. I want to do it in Python.

Thank you so much for this great project and I'm really excited of the further results :)

When I'm trying to test keras-simple project on my virtualenv activated in the directory, it cannot import keras module installed in the virtualenv but my REPL is unexpectedly killed instead.

I think this is really important support since many of python developers prefer to work in virtualenv. Is there any way to do it or do you have a plan for this functionality?

Thanks again for your work.

expected:

(python/str \a) ;;=> "a"

actual: 💥

Unable to convert objects of type: class java.lang.Character

(require '[clojure.core.async :reload true]) 
(require-python '[math :reload])

We should change the syntax to be consistent (and do it before the syntax percolates into code too far)

First of all, THANK YOU for building this. It's exactly what I've wanted for so long!

I'm trying to use ta-lib in my REPL to pull in various technical indicators and use matplotlib to chart the series. It works 90% of the time, but every tenth time or so, my application crashes with the below stack trace. Any idea what I should be avoiding or what can cause this? Many thanks!

# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007f5a1f876211, pid=635, tid=709
#
# JRE version: OpenJDK Runtime Environment (11.0.6+10) (build 11.0.6+10)
# Java VM: OpenJDK 64-Bit Server VM (11.0.6+10, mixed mode, tiered, compressed oops, g1 gc, linux-amd64)
# Problematic frame:
# C  [libpython3.7m.so+0x276211]  _PyObject_FastCallDict+0x71
#
# No core dump will be written. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# An error report file with more information is saved as:
# /app/hs_err_pid635.log
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.java.com/bugreport/crash.jsp
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.

Clojure 1.10.1
user=> (ns user (:require [libpython-clj.python :as py]))
Execution error (NullPointerException) at tech.jna.base/fn (base.clj:35).
null

I'm use python 3.7 and compile project as lein uberjar.

Thanks.

This avoids issues like:

lein test :only panthera.generics-test/head

ERROR in (head) (interpreter.clj:385)
expected: (= (u/->clj (g/head (g/data-frame (flatten (repeat 5 [{:a 1, :b 2} {:a 2, :
  actual: java.lang.Exception: Traceback (most recent call last):
  File "/mnt/DATA/miniconda3/envs/panthera/lib/python3.7/site-packages/pandas/core/fr
    elif isinstance(data, abc.Iterable) and not isinstance(data, (str, bytes)):
  File "/mnt/DATA/miniconda3/envs/panthera/lib/python3.7/abc.py", line 139, in __inst
    return _abc_instancecheck(cls, instance)
  File "/mnt/DATA/miniconda3/envs/panthera/lib/python3.7/abc.py", line 143, in __subc
    return _abc_subclasscheck(cls, subclass)
TypeError: issubclass() arg 1 must be a class

• PyCall.jl pywith

@pywith pybuiltin("open")("file.txt","w") as f begin
    f.write("hello")
end

Hello,

Thanks a lot for the library, it has been a real pleasure to use.

For now, we can call a function like this:

(py/$a np linspace :start 0 :stop_button:  1)

but it would be great if we could integrate a macro, say $a** such that we could perform

(py/$a** np linspace {:start 0 :stop 1})

Obviously we could also do this

(defn $a** [x attr m] (py/call-attr-kw x attr nil m))

However, I think it would be a beneficial addition to the library.

Best regards,
David

A friend who uses python regularly indicated to me

"When it comes to python, sandboxed dependency environments are a must."

To that end I am able to do something like:

brew install pipenv
pipenv install seaborn
pipenv shell
python

then run the examples here: https://seaborn.pydata.org/introduction.html

However, the clojure seaborn example will error out for me at this line: https://github.com/gigasquid/libpython-clj-examples/blob/master/src/gigasquid/seaborn.clj#L17

with

gigasquid.seaborn=> gigasquid.seaborn=> (require-python '[seaborn :as sns])
:ok
gigasquid.seaborn=> (require-python '[matplotlib.pyplot :as pyplot])
:ok
gigasquid.seaborn=> (sns/set)
nil
gigasquid.seaborn=> (def dots (sns/load_dataset "dots"))
Execution error at libpython-clj.python.interpreter/check-error-throw (interpreter.clj:474).
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 1317, in do_open
    encode_chunked=req.has_header('Transfer-encoding'))
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 1229, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 1275, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 1224, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 1016, in _send_output
    self.send(msg)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 956, in send
    self.connect()
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 1392, in connect
    server_hostname=server_hostname)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/ssl.py", line 412, in wrap_socket
    session=session
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/ssl.py", line 850, in _create
    self.do_handshake()
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/ssl.py", line 1108, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1045)

Is there a way to make it use the correct python, i.e. the one in the pipenv shell?

Many thanks

In Python, object.method(*args, **kwargs) is equivalent to class.method(object, *args, **kwargs).

From a user perspective, the class-method-object syntax is usually indistinguishable from module-function syntax, aka, math.sin(1). A concrete example:

>>> l = [1, 2, 3]
>>> list.append(l, 4)
>>> l
[1, 2, 3, 4]

could just as easily be accomplished in libpython-clj via

(def l (python/list [1 2 3]))
(python.list/append 4)
l ;;=> [1, 2, 3, 4]

This may require special corresponding methods such as py/set-ns-class-attr! if mutating class attributes are necessary. This may also require additionally bookkeeping in require-python to make sure that when a namespace is reloaded, all corresponding ns-classes are also reloaded.

As discussed here.

We now have more or less generalized binding for python to datafy/nav. We think, more or less :-).

If someone is so inclined, some testing/tutorials around annotating some python library and seeing what happens when they try to datafy/nav it via REBL would be cool.

https://clojure.org/reference/reader#_symbols says:

Symbols beginning or ending with '.' are reserved by Clojure

In particular clojure uses them as syntax sugar for (new Whatever ...), this library defines several macros with names that end in '.' that for now happen to work as clojure itself doesn't enforce this reservation, but if it ever does it will break this library. And other tooling that respects this reservation breaks with this library

When working with libpython-clj from other projects or from journals like https://nextjournal.com/kommen/gigasquids-libpython-clj-examples, it's nice to be able to use a :git/url and sha to pin a specific commit - and not have to wait for a maven deployed version, (or have to clone the repo and install yourself.)

I'm happy to pitch in and look into converting to clojure deps and make it work with Circle CI if there is interest in this.

Attempted to use this with mxnet today. Here are thoughts so far:

It worked very well and provided good documentation and discoverability. This is awesome and a solid pathway forward.

1. The biggest thing is the partial pathways have to work. We can't always load everything recursively from the root down dynamically due to time constraints or just various constraints. Some modules/classes can't be loaded all the time.
2. It has to work on submodules just as classes in a module. In mxnet, for example, the submodules are attributes of the main mxnet module but you can't import them. You have to access them via the '.' operator.
3. It is odd that I can (require-python '[builtins :alpha-load-ns-classes :as python]) but I cannot (require-python '[builtins.list :as python.list]). This isn't pure Clojure syntax but it may be the best way.

For instance, in mxnet, modules is a dynamic module beneath mxnet and it has a Module class with important methods on it. I want to type (mxnet.modules.Module/forward item batch-data) and have intellisense help me out. This last point I think is actually the most important. I think we should be able to 'require' python classes as long as they are static variables of the python module. Ditto submodules.

In order to achieve some of the above in a consistent manner, I think we update the system so that it attempts to require the root module and then walk down using getattr to the target module. We then create the namespace and go from there. We don't assume the input path to require is a module itself but we do assume we can access it at runtime statically via get-attr calls.

I think also :alpha-load-ns-classes should just be renamed :recurse and there should be a list of exclusions :recurse-exclude that filters particularly expensive or crashy things out of the recursive pathway.

With these 2 changes and some solid testing I think this moves us another measurable step forward in terms of pure language integration.

I have tried running the example on osx.
The way to do that was:
sudo find / -name libpython3.6m.dylib
And taking the first results:
(alter-var-root #'libpython-clj.jna.base/python-library (constantly "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/config-3.6m-darwin/libpython3.6m.dylib"))

I was wondering if there is an easy way to get the current python based on the context (virtualenv, conda) and figure out the location of the libpython?

Hi @cnuernber,
thank you for the great library. I just wonder whether is it possible to utilize type checking for Python types. Unfortunately, I haven't found any reference in a source code.
Thanks in advance

Currently the Python3 executable path is hardcoded to "python3". I propose we allow the executable to be specified via :python-executable in py/initialize!. On my system, Python3.6 always gets picked up first even though Python3.7 is available because Python3.6 is my current system alternative.

This change should also allow use of Python virtual environments.

Currently load-python-lib is very procedural and therefore difficult to test. I suggest refactoring to use a pattern of a series of functions which build up a map of information, followed by a series of procedures which take action on that map.

If the side-effecty procedures which take actions on the load-python-lib-map are small and well understood, it will make debugging easier and confidence higher.

Similarly, the map produced by the series of data transformations in the preceding step will be able to be tested.

Once again, amazing project!

I was experimenting if it's possible to use flax with libpython-clj. I'm pretty much asking for trouble by combining these early-stage libraries, but it's just for fun. It seems that the code below fails because the function apply that libpython-clj defines on the fly does not have a Function Signature Object associated with it, which flax tries to use via inspect.signature. Or I'm missing something in my Clojure code?

Just wondering if this is a known limitation, or there is a way to include this signature? Feel free to close the issue if this is not a top priority.

(ns flax_model
  (:require [libpython-clj.require :refer [require-python]]
            [libpython-clj.python :as py]]))

(require-python '[flax.nn :as nn])

(def identity (py/create-class "Identity" [nn/Module]
                               {"apply" (py/make-tuple-instance-fn 
                                         (fn [self x] x)
                                         :arg-converter py/as-jvm
                                         :method-name "apply")}))

Stack trace:

Execution error at libpython-clj.python.interpreter/check-error-throw (interpreter.clj:474).
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/flax/nn/base.py", line 169, in __init__
    apply_params = _fn_parameters(apply_fn)
  File "/usr/local/lib/python3.7/site-packages/flax/nn/base.py", line 153, in _fn_parameters
    return tuple(inspect.signature(fn).parameters.values())
  File "/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/lib/python3.7/inspect.py", line 3083, in signature
    return Signature.from_callable(obj, follow_wrapped=follow_wrapped)
  File "/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/lib/python3.7/inspect.py", line 2833, in from_callable
    follow_wrapper_chains=follow_wrapped)
  File "/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/lib/python3.7/inspect.py", line 2288, in _signature_from_callable
    skip_bound_arg=skip_bound_arg)
  File "/usr/local/Cellar/python/3.7.4/Frameworks/Py
thon.framework/Versions/3.7/lib/python3.7/inspect.py", line 2112, in _signature_from_builtin
    raise ValueError("no signature found for builtin {!r}".format(func))
ValueError: no signature found for builtin <built-in function apply>

class clojure.lang.Compiler$CompilerException

I can't load the library, running in a REPL.

12:13:00 ~/Workspaces/clojure/cljfx-play $  
→ python --version
Python 3.8.0
12:13:02 ~/Workspaces/clojure/cljfx-play $  
→ java --version
openjdk 11.0.6 2020-01-14
OpenJDK Runtime Environment AdoptOpenJDK (build 11.0.6+10)
OpenJDK 64-Bit Server VM AdoptOpenJDK (build 11.0.6+10, mixed mode)
12:14:10 ~/Workspaces/clojure/cljfx-play $  
→ clj
Clojure 1.10.1
user=> (require '[libpython-clj.require :refer [require-python]])
Jan. 24, 2020 12:07:47 PM clojure.tools.logging$eval494$fn__497 invoke
INFO: Executing python initialize with options:{:python-executable nil, :program-name nil, :python-home nil, :library-path nil}
Jan. 24, 2020 12:07:47 PM clojure.tools.logging$eval494$fn__497 invoke
INFO: Detecting startup-info for Python executable: 
Jan. 24, 2020 12:07:47 PM clojure.tools.logging$eval494$fn__497 invoke
INFO: Startup info detected: {:python-home "/Users/stuartrexking/.pyenv/versions/3.8.0", :lib-version "3.8", :libname "python3.8m", :java-library-path-addendum "/Users/stuartrexking/.pyenv/versions/3.8.0/lib"}
Jan. 24, 2020 12:07:47 PM clojure.tools.logging$eval494$fn__497 invoke
INFO: Setting java library path: /Users/stuartrexking/.pyenv/versions/3.8.0/lib:/Users/stuartrexking/Library/Java/Extensions:/Library/Java/Extensions:/Network/Library/Java/Extensions:/System/Library/Java/Extensions:/usr/lib/java:.
Jan. 24, 2020 12:07:47 PM clojure.tools.logging$eval494$fn__497 invoke
INFO: Reference thread starting
Syntax error (ExceptionInfo) compiling at (libpython_clj/metadata.clj:13:1).
Failed to load library
user=> 

clj -A:new libpython-clj myapp/myname

would be pretty awesome!

But as a start,

clj -A:new https://github.com/cnuernber/libpython-clj@<sha> mydomain/myapp

would be good too!

Reported here: https://clojurians.zulipchat.com/#narrow/stream/215609-libpython-clj-dev/topic/pandas.20query.20methods.20bug/near/186054336

(require-python
 '([pandas :as pd]
   [numpy :as np]))

pd  ;; outputs the python outputs in 1.30
np ;;  Error -> Unable to resolve symbol: os in this context in 1.31

Per comment from community member it appears that require-python's use of :all might not match Clojure's use of :all. i.e., it seems that the behavior of :all keeps the interned symbols private. I think it's important (for the sake of library authors and symbol exporting) to be able to make those public but it should probably use a different syntax. If we're going to play in Clojure land we need to follow Clojure rules!

Currently there is a mismatch between require-python syntax and require syntax.

(require '(a b c)) implies (require 'a.b 'a.c).

Currently, require-python syntax operates in the following manner:

(require-python '(a b c)) implies

(do 
  (require-python 'a)
  (require-python 'b)
  (require-python 'c))

This is incorrect.

(require-python '(a b c)) should also imply (require-python 'a.b 'a.c) (which also doesn't work), and (require-python 'a.b 'a.c) should imply

(do 
  (require-python 'a.b)
  (require-python 'a.c))

Special attention should also be paid to allow for (require-python '(a [b :as c :refer [blah]])) etc
which should imply (require-python '[a.b :as c :refer [blah]]).

Thanks for this project! I enjoyed the talk from this year's Conj and it made me want to dive right in.

My use case is somewhat unique, I think: I'm trying to use Python to drive some hardware I have soldered to the GPIO pins on a Raspberry Pi Zero. The Pi is a 32-bit ARM processor, and Python works great on it... but Python interop has been giving me a bit of trouble.

Here's what we're working with:

 $ ldconfig -p | grep python
        libpython3.7m.so.1.0 (libc6,hard-float) => /usr/lib/arm-linux-gnueabihf/libpython3.7m.so.1.0
        libpython3.7m.so (libc6,hard-float) => /usr/lib/arm-linux-gnueabihf/libpython3.7m.so
        libpython2.7.so.1.0 (libc6,hard-float) => /usr/lib/arm-linux-gnueabihf/libpython2.7.so.1.0
        libpython2.7.so (libc6,hard-float) => /usr/lib/arm-linux-gnueabihf/libpython2.7.so

and Java:

$ java -version
openjdk version "1.8.0_152"
OpenJDK Runtime Environment (Zulu Embedded 8.25.0.76-linux-aarch32hf) (build 1.8.0_152-b76)
OpenJDK Client VM (Zulu Embedded 8.25.0.76-linux-aarch32hf) (build 25.152-b76, mixed mode, Evaluation)

I'm using Python 3.7 to drive the hardware, and Python itself runs fine.
Likewise, I can run Clojure (both JARs and Clojure in the REPL) without any issue.
But when I try to run Python from within Clojure via the REPL, here's what happens:

=> (require '[libpython-clj.python :as py])
nil
=> (py/initialize!)
Jan 09, 2020 3:05:16 AM clojure.tools.logging$eval1906$fn__1909 invoke
INFO: Executing python initialize!
Jan 09, 2020 3:05:17 AM clojure.tools.logging$eval1906$fn__1909 invoke
INFO: Startup info detected: {:python-home "/usr", :lib-version "3.7", :libname "python3.7m", :java-library-path-addendum "/usr/lib"}
Jan 09, 2020 3:05:17 AM clojure.tools.logging$eval1906$fn__1909 invoke
INFO: Reference thread starting
Jan 09, 2020 3:05:18 AM clojure.tools.logging$eval1906$fn__1909 invoke
INFO: Library python3.7m found at [:system "python3.7m"]
Execution error (IllegalArgumentException) at libpython-clj.jna.base/eval22533$fn$G (base.clj:20).
No implementation of method: :->py-object-ptr of protocol: #'libpython-clj.jna.base/PToPyObjectPtr found for class: nil

The error repeats itself if you try any other commands:

=> (py/run-simple-string "print('Hello')")
Execution error (IllegalArgumentException) at libpython-clj.jna.base/eval22533$fn$G (base.clj:20).
No implementation of method: :->py-object-ptr of protocol: #'libpython-clj.jna.base/PToPyObjectPtr found for class: nil

I started the repl using lein repl in a new project root.
The project was scaffolded with the lein new app command.

Do you see anything off the top of your head that may be the root cause of these issues?

My gut tells me it's one of three things:

1. The 32-bit ARM processor, and everything that goes along with that
2. The Zulu Embedded Java 8 JDK
3. Something dumb that I haven't noticed yet.

Any thoughts as to why this is happening, or is there any more information I could provide?

Really appreciate your time. Thank you!

Expected:

(try
    (throw python/Exception)
    (catch python/Exception e 
      :win)) ;;=> :win 

Actual: 💥

Hello, I am learning about ML and trying to port https://github.com/pytorch/examples/blob/master/mnist/main.py
to libpython-clj since I love Clojure.

Quickly scanning through the code and documentation, I do not see any way to subclass the torch.nn.Module into your own neural net from the Clojure side. Am I missing something, or is there an example somewhere?

Uberjar's using require-python are currently failing.

We at least need to dig in, investigate, and document how to work around this if not fix it.

Hello,

First of all, this is a great project and I wish we could see it being the standard way to interact with python and Clojure!

It is only a small detail with respect to the all difficulties you might have for creating a bridge between python and Clojure, but would it make sense to create some sort of sugar/interface/macro to call functions from modules with would look like either namespace calling, e.g np/zeros for modules or like (.attr object & args) or
(.attr object & kwargs) for modules and object.

The libpython system was built with incomplete support for multiple interpreters.

Filling out this support will allow one to control python from multiple threads and use some serializable communication format (like pickle or json) to communicate python objects between interpreters.

The rules for libpython w/r/t threading are fairly clearly written on their threading API page. Interpreters of some scope of memory allocated to them and their threading primitives include at least:

1. save interpreter state to variable and release lock.
2. load interpreter state from variable and acquire lock.
3. swap active interpreter state in the current thread which assumes the lock is acquired.

libpython-clj's behavoir w/r/t all this is somewhat summarized here.

The gist is this - libpython binds the current interpreter to a thread-local variable when it acquires the gil. If one is bound then it attempts a swap. It does this with a release on try...finally. It also uses Java synchronization to make sure that interpreters are not reentrant in some unexpected way.

So, the work required here is to, using the libc bindings, build out the pathway further for multiple interpreters. A rough outline may be:

1. (defn create-interpreter ...)
2. check that two interpreters can execute serially first, then simultaneously in multiple threads.
3. Attempt some limited communication between them.
4. Test and ensure bridging works appropriately between them.

My opinion is that 1,2 would be useful enough to be a clear PR. Then we can carry it forward with 3 and 4 in separate stages being careful to check and be correct as we can w/r/t memory and threading semantics at each step. I also believe this is an important and necessary in the growth of this library.

The relevant portions of the zulip chat that prompted this are pasted below.

I was thinking about it, but would it be possible to create multiple python interpreters/processes in a single/multiple namespaces? If yes, we could sell it as a solution for concurrency in python :) That being said it will be interesting to be able to provide explicitly an interpreter/python process to a module, or the ability to explicitly attach a module to an interpreter.

It would also means we could have isolation of data in the data analysis processes avoid namespace pollution all too common in python.

(By the way, not being able to do that seems to be one of the limitations of Reticulate, the R package for Python interop:
https://stackoverflow.com/questions/50762140/r-reticulate-how-can-i-close-restart-the-python-console
Hearing from people who use it intensively, that is indeed a practical problem. In cases of long running computations, as well as of memory leaks, being able to get rid of a python process and create a new one would be useful.)

Provide support so that users and library authors can define the behavior of datafy and nav for Python objects.

Hello Chris!

I really enjoyed listening to Scicloj meeting #4. You're doing interesting work here. I feel that you're setting the Clojure community up for an interop story with Python that's similar to how we're able to make use of Java.

I have a question, if you don't mind. Per now, I can use my own interpreter of choice like this:

(require '[libpython-clj.python :as py]
         '[libpython-clj.jna.base])

(defn get-python [_]
  "/home/teodorlu/opt/anaconda3/bin/python")

(alter-var-root #'libpython-clj.jna.base/*python-library* #'get-python)

I would like to use libpython-clj to interface with a running Python process that's not started from within the Clojure process, but "on its own terms". A possible process could look like this:

1. Make some changes to the remote Python interpreter allow libpython-clj to ... do what it needs to do.
2. Start a Clojure process
3. Point the Clojure process to the running Python process
4. Evaluate Python code in that context.

The reason why I can't just start a new process, is that the Python process is started by another application, and I need access to the application state, as the user interacts with the state.

• Is this in scope for libpython-clj?
• Do you think it's feasible?

I spent some time reading the libpython-clj source code, but I must admit that this is above my level, hence I'm asking.

Thanks again for your work, and have a nice day!
Teodor

Sequences (1.24+) are breaking how panthera works, basically this works:

(pt/data-frame (vec (flatten (repeat 5 [{:a 1 :b 2} {:a 2 :b 3}]))))

But this doesn't:

(pt/data-frame (flatten (repeat 5 [{:a 1 :b 2} {:a 2 :b 3}])))

Execution error at libpython-clj.python.interpreter/check-error-throw (interpreter.clj:506).
Traceback (most recent call last):
  File "/mnt/DATA/miniconda3/envs/panthera/lib/python3.7/site-packages/pandas/core/frame.py", line 443, in __init__
    elif isinstance(data, abc.Iterable) and not isinstance(data, (str, bytes)):
  File "/mnt/DATA/miniconda3/envs/panthera/lib/python3.7/abc.py", line 139, in __instancecheck__
    return _abc_instancecheck(cls, instance)
  File "/mnt/DATA/miniconda3/envs/panthera/lib/python3.7/abc.py", line 143, in __subclasscheck__
    return _abc_subclasscheck(cls, subclass)
TypeError: issubclass() arg 1 must be a class

Just as a reference, this breaks in the same way:

(require-python '[pandas :as pd])
(pd/DataFrame (flatten (repeat 5 [{:a 1 :b 2} {:a 2 :b 3}])))

I'm already sanitizing user input a lot (https://github.com/alanmarazzi/panthera/blob/e16d254d7d8048af0381cbf2e3285a77db517fb2/src/panthera/pandas/utils.clj#L229) because pandas expects fully realized objects shaped in a certain way to actually work.

P.S.: refer to the dev branch since there's a lot of stuff that isn't in master yet

From the slack channel

is it possible in the clojure machinery to do :refer-clojure :exclude :all: in the generated namespaces
the warnings about "java.lang.Exception is being overridden" are a bit much

We need help with this. It is possible, it is a matter of figuring out the changes that conda makes to the system and what env variables the conda python shim sets up.

This is very important for newcomer experience so if you know anyone who knows how conda works or want to spend some time working with conda and libpython-clj please jump right in.

Hi,

I have a question regarding python array traversing.
What is the fastest way to traverse python array? The fastest variant I came up with:

(arr/array
         array-type
         (with-gil
                 (->> (range (libpy/PySequence_Length samples))
                      (mapv (fn [idx]
                              (let [time (/ idx frame-rate)
                                    gain (* scale
                                            (Math/sin (* twoPiF time)))]
                                (int (* gain
                                        (-> (libpy/PySequence_GetItem samples idx)
                                            wrap-pyobject
                                            ->jvm)))))))))

It is inefficient as I initialize a vector from input python array and after that converting that vector back to the python array.
According to benchmark, converting back to python array takes equal time that is needed to initialize vector from input.

So maybe there is some kind of better way iterating over python array and simultaneously
traversing its elements, so I can get a traversed version of an array as output, but unfortunately I couldn't figure out how to to do that.

Could you please help me to come up with better solution?

Thank you in advance for your effort and time

Hi!

Thank you so much for this awesome library. We are currently experimenting with it and it seems to deliver big advantages for us compared to RPC through TCP/Protobuf that we have used before communicating between python and clojure-processes.

I hit a very odd issue on MacOS X, with Python 3.7.6_1 with libpython python3.7m.

If I require the sys module as the first requirement, it fails with a message:

maharj:javascript $ clj -Sdeps '{:deps {clj-python/libpython-clj {:mvn/version "1.36"}}}'
Clojure 1.10.1
user=> (require '[libpython-clj.require :as pyr])
Feb 20, 2020 8:50:41 AM clojure.tools.logging$eval496$fn__499 invoke
INFO: Executing python initialize with options:{:python-executable nil, :program-name nil, :python-home nil, :library-path nil}
Feb 20, 2020 8:50:41 AM clojure.tools.logging$eval496$fn__499 invoke
INFO: Detecting startup-info for Python executable:
Feb 20, 2020 8:50:41 AM clojure.tools.logging$eval496$fn__499 invoke
INFO: Startup info detected: {:python-home "/usr/local/Cellar/python/3.7.6_1/Frameworks/Python.framework/Versions/3.7", :lib-version "3.7", :libname "python3.7m", :java-library-path-addendum "/usr/local/Cellar/python/3.7.6_1/Frameworks/Python.framework/Versions/3.7/lib"}
Feb 20, 2020 8:50:41 AM clojure.tools.logging$eval496$fn__499 invoke
INFO: Setting java library path: /usr/local/Cellar/python/3.7.6_1/Frameworks/Python.framework/Versions/3.7/lib:/Users/maharj/Library/Java/Extensions:/Library/Java/Extensions:/Network/Library/Java/Extensions:/System/Library/Java/Extensions:/usr/lib/java:.
Feb 20, 2020 8:50:41 AM clojure.tools.logging$eval496$fn__499 invoke
INFO: Reference thread starting
Feb 20, 2020 8:50:41 AM clojure.tools.logging$eval496$fn__499 invoke
INFO: Library python3.7m found at [:java-library-path "/usr/local/Cellar/python/3.7.6_1/Frameworks/Python.framework/Versions/3.7/lib/libpython3.7m.dylib"]
Feb 20, 2020 8:50:41 AM clojure.tools.logging$eval496$fn__499 invoke
INFO: Library c found at [:system "c"]
nil
user=> (pyr/require-python 'sys)
Execution error (ExceptionInfo) at libpython-clj.jna.base/ensure-pyobj (base.clj:52).
Failed to get a pyobject pointer from object.
user=> (pyr/require-python 'sys)
:ok

However, if I import another module, say os before, it works like expected.

I did not have time to inspect this more at this time, but as I've stepped through the code with debugger, the problem disappears on some occasion, so maybe this is due to some timing issue? It does not, however correllate with the time one waits between executing the (pyr/require 'sys) after requiring the library (as it does its initialization on the start of the load, one could expect that it would have some effect...).

Thanks!

• The KeyedVectors is class, How to import it ?

from gensim.models.keyedvectors import KeyedVectors

• https://github.com/cnuernber/libpython-clj/blob/master/example/src/matplotlib.clj

matplotlib-clj.core=> (plot-it)
2019-11-15 21:02:28.865 java[21046:1140194] WARNING: NSWindow drag regions should only be invalidated on the Main Thread! This will throw an exception in the future. Called from (
	0   AppKit                              0x00007fff2c69e6ef -[NSWindow(NSWindow_Theme) _postWindowNeedsToResetDragMarginsUnlessPostingDisabled] + 371
	1   AppKit                              0x00007fff2c69bbc1 -[NSWindow _initContent:styleMask:backing:defer:contentView:] + 1416
	2   AppKit                              0x00007fff2c69b633 -[NSWindow initWithContentRect:styleMask:backing:defer:] + 42
	3   _macosx.cpython-37m-darwin.so       0x00000001361646b0 -[Window initWithContentRect:styleMask:backing:defer:withManager:] + 80
	4   _macosx.cpython-37m-darwin.so       0x0000000136167bc7 FigureManager_init + 327
	5   Python                              0x000000012ddba050 wrap_init + 12
	6   Python                              0x000000012dd81407 wrapperdescr_call + 337
	7   Python                              0x000000012dd7b668 _PyObject_FastCallKeywords + 365
	8   Python                              0x000000012de123ad call_function + 746
	9   Python                              0x000000012de0af6f _PyEval_EvalFrameDefault + 7020
	10  Python                              0x000000012dd7bbdd function_code_fastcall + 112
	11  Python                              0x000000012dd7c561 _PyObject_Call_Prepend + 150
	12  Python                              0x000000012ddb9fbe slot_tp_init + 80
	13  Python                              0x000000012ddb6c85 type_call + 178
	14  Python                              0x000000012dd7b668 _PyObject_FastCallKeywords + 365
	15  Python                              0x000000012de123ad call_function + 746
	16  Python                              0x000000012de0af6f _PyEval_EvalFrameDefault + 7020
	17  Python                              0x000000012dd7bbdd function_code_fastcall + 112
	18  Python                              0x000000012de123b4 call_function + 753
	19  Python                              0x000000012de0af6f _PyEval_EvalFrameDefault + 7020
	20  Python                              0x000000012de12c5c _PyEval_EvalCodeWithName + 1857
	21  Python                              0x000000012dd7b42a _PyFunction_FastCallDict + 441
	22  Python                              0x000000012dd7c561 _PyObject_Call_Prepend + 150
	23  Python                              0x000000012dd7b910 PyObject_Call + 136
	24  Python                              0x000000012de0b286 _PyEval_EvalFrameDefault + 7811
	25  Python                              0x000000012de12c5c _PyEval_EvalCodeWithName + 1857
	26  Python                              0x000000012dd7b7c7 _PyFunction_FastCallKeywords + 225
	27  Python                              0x000000012de123b4 call_function + 753
	28  Python                              0x000000012de0b00d _PyEval_EvalFrameDefault + 7178
	29  Python                              0x000000012dd7bbdd function_code_fastcall + 112
	30  Python                              0x000000012de123b4 call_function + 753
	31  Python                              0x000000012de0b00d _PyEval_EvalFrameDefault + 7178
	32  Python                              0x000000012de12c5c _PyEval_EvalCodeWithName + 1857
	33  Python                              0x000000012dd7b7c7 _PyFunction_FastCallKeywords + 225
	34  Python                              0x000000012de123b4 call_function + 753
	35  Python                              0x000000012de0b00d _PyEval_EvalFrameDefault + 7178
	36  Python                              0x000000012de12c5c _PyEval_EvalCodeWithName + 1857
	37  Python                              0x000000012dd7b42a _PyFunction_FastCallDict + 441
	38  jna2771800009792166730.tmp          0x0000000127a9be74 ffi_call_unix64 + 76
	39  ???                                 0x000070000ded2548 0x0 + 123145535956296
)

Commit edd221d appears to have removed the def for bridged:

(def bridged (->> *1
                        (map (fn [[k v]]
                               [k (as-jvm v)]))
                        (into {})))

That makes the current code in the README:

(instance? java.util.Map (:globals bridged))

not work.

Not all python libraries tolerate multithreaded access well, even when serialized by a GIL. Some, for instance, use static/thread-local variables in a way as to make then unsafe to use from multiple threads.

We want to make purely single threaded access possible when necessary for robustness. It should be possible to load python and use it from an agent and guarantee all python access is from one thread.

One potential soution is a custom gc release-refcount queue that is cleared at the beginning or end of with-gil.