Comments (14)
cnuernber
commented on July 19, 2024
Hey, thanks, an embedded use case is an interesting pathway for sure. The entire stack trace would be helpful. That method gets called from a lot of potential places.
On line 48 there is ensure-pyobj - I think something is nil in there and thus you are getting the exception but we need to figure out what part of the initialization process is failing. So a full stack trace of the error would be helpful.
It looks as if the first half of initialize happens correctly. The second half is when it is overriding stdout/stderr. One thing you may try is to request the system not attempt io redirection as this simplifies the initialization process (but you won't get stderr/stdout reflected into out and err).
https://github.com/cnuernber/libpython-clj/blob/master/src/libpython_clj/python.clj#L229
Aside from that, we need to get full stack traces :-).
from libpython-clj.
ajpierce
commented on July 19, 2024
Hi, Chris! Thanks for the prompt reply.
I don't yet have the pleasure of being able to use Clojure at my day job, so I was pleasantly surprised to learn about the *e variable tonight after seeing your message! I had no idea that we could extract more from the REPL when things go awry, so thank you for that 😁
=> (require '[libpython-clj.python :as py])
nil
=> (py/initialize!)
Jan 10, 2020 2:49:19 AM clojure.tools.logging$eval1906$fn__1909 invoke
INFO: Executing python initialize!
Jan 10, 2020 2:49:20 AM clojure.tools.logging$eval1906$fn__1909 invoke
INFO: Startup info detected: {:python-home "/usr", :lib-version "3.7", :libname "python3.7m", :java-library-path-addendum "/usr/lib"}
Jan 10, 2020 2:49:20 AM clojure.tools.logging$eval1906$fn__1909 invoke
INFO: Reference thread starting
Jan 10, 2020 2:49:21 AM clojure.tools.logging$eval1906$fn__1909 invoke
INFO: Library python3.7m found at [:system "python3.7m"]
Execution error (IllegalArgumentException) at libpython-clj.jna.base/eval22533$fn$G (base.clj:20).
No implementation of method: :->py-object-ptr of protocol: #'libpython-clj.jna.base/PToPyObjectPtr found for class: nil
=> *e
#error {
:cause "No implementation of method: :->py-object-ptr of protocol: #'libpython-clj.jna.base/PToPyObjectPtr found for class: nil"
:via
[{:type java.lang.IllegalArgumentException
:message "No implementation of method: :->py-object-ptr of protocol: #'libpython-clj.jna.base/PToPyObjectPtr found for class: nil"
:at [clojure.core$_cache_protocol_fn invokeStatic "core_deftype.clj" 583]}]
:trace
[[clojure.core$_cache_protocol_fn invokeStatic "core_deftype.clj" 583]
[clojure.core$_cache_protocol_fn invoke "core_deftype.clj" 575]
[libpython_clj.jna.base$eval22533$fn__22534$G__22524__22539 invoke "base.clj" 20]
[libpython_clj.jna.base$ensure_pyobj invokeStatic "base.clj" 50]
[libpython_clj.jna.base$ensure_pyobj invoke "base.clj" 48]
[libpython_clj.jna.concrete.unicode$PyUnicode_AsUTF8 invokeStatic "unicode.clj" 79]
[libpython_clj.jna.concrete.unicode$PyUnicode_AsUTF8 invoke "unicode.clj" 79]
[libpython_clj.jna.interpreter$get_type_name invokeStatic "interpreter.clj" 219]
[libpython_clj.jna.interpreter$get_type_name invoke "interpreter.clj" 216]
[clojure.lang.Var invoke "Var.java" 384]
[libpython_clj.python.interpreter$py_type_keyword$fn__24162 invoke "interpreter.clj" 340]
[clojure.lang.Atom swap "Atom.java" 37]
[clojure.core$swap_BANG_ invokeStatic "core.clj" 2352]
[clojure.core$swap_BANG_ invoke "core.clj" 2345]
[libpython_clj.python.interpreter$py_type_keyword invokeStatic "interpreter.clj" 333]
[libpython_clj.python.interpreter$py_type_keyword invoke "interpreter.clj" 327]
[libpython_clj.python.object$eval26422$fn__26423$fn__26424 invoke "object.clj" 231]
[libpython_clj.python.interpreter$with_gil_fn invokeStatic "interpreter.clj" 377]
[libpython_clj.python.interpreter$with_gil_fn invoke "interpreter.clj" 346]
[libpython_clj.python.object$eval26422$fn__26423 invoke "object.clj" 230]
[libpython_clj.python.protocols$eval24268$fn__24269$G__24259__24274 invoke "protocols.clj" 21]
[libpython_clj.python.protocols$python_type invokeStatic "protocols.clj" 30]
[libpython_clj.python.protocols$python_type invoke "protocols.clj" 26]
[libpython_clj.python.object$wrap_pyobject invokeStatic "object.clj" 136]
[libpython_clj.python.object$wrap_pyobject doInvoke "object.clj" 122]
[clojure.lang.RestFn invoke "RestFn.java" 410]
[libpython_clj.python.object$incref_wrap_pyobject$fn__26409 invoke "object.clj" 189]
[libpython_clj.python.interpreter$with_gil_fn invokeStatic "interpreter.clj" 377]
[libpython_clj.python.interpreter$with_gil_fn invoke "interpreter.clj" 346]
[libpython_clj.python.object$incref_wrap_pyobject invokeStatic "object.clj" 186]
[libpython_clj.python.object$incref_wrap_pyobject invoke "object.clj" 182]
[libpython_clj.python.interop$add_module$fn__26904 invoke "interop.clj" 87]
[libpython_clj.python.interpreter$with_gil_fn invokeStatic "interpreter.clj" 377]
[libpython_clj.python.interpreter$with_gil_fn invoke "interpreter.clj" 346]
[libpython_clj.python.interop$add_module invokeStatic "interop.clj" 85]
[libpython_clj.python.interop$add_module invoke "interop.clj" 83]
[libpython_clj.python.interop$register_bridge_type_BANG_$fn__26947 invoke "interop.clj" 281]
[libpython_clj.python.interpreter$with_gil_fn$fn__24167$fn__24168 invoke "interpreter.clj" 360]
[clojure.lang.AFn applyToHelper "AFn.java" 152]
[clojure.lang.AFn applyTo "AFn.java" 144]
[clojure.core$apply invokeStatic "core.clj" 665]
[clojure.core$with_bindings_STAR_ invokeStatic "core.clj" 1973]
[clojure.core$with_bindings_STAR_ doInvoke "core.clj" 1973]
[clojure.lang.RestFn invoke "RestFn.java" 425]
[libpython_clj.python.interpreter$with_gil_fn$fn__24167 invoke "interpreter.clj" 358]
[clojure.lang.AFn applyToHelper "AFn.java" 152]
[clojure.lang.AFn applyTo "AFn.java" 144]
[clojure.core$apply invokeStatic "core.clj" 665]
[clojure.core$with_bindings_STAR_ invokeStatic "core.clj" 1973]
[clojure.core$with_bindings_STAR_ doInvoke "core.clj" 1973]
[clojure.lang.RestFn invoke "RestFn.java" 425]
[libpython_clj.python.interpreter$with_gil_fn invokeStatic "interpreter.clj" 356]
[libpython_clj.python.interpreter$with_gil_fn invoke "interpreter.clj" 346]
[libpython_clj.python.interop$register_bridge_type_BANG_ invokeStatic "interop.clj" 280]
[libpython_clj.python.interop$register_bridge_type_BANG_ doInvoke "interop.clj" 276]
[clojure.lang.RestFn invoke "RestFn.java" 397]
[libpython_clj.python$initialize_BANG_ invokeStatic "python.clj" 244]
[libpython_clj.python$initialize_BANG_ doInvoke "python.clj" 229]
[clojure.lang.RestFn invoke "RestFn.java" 397]
[squishy_kitty.core$eval28099 invokeStatic "form-init4165223302715120006.clj" 1]
[squishy_kitty.core$eval28099 invoke "form-init4165223302715120006.clj" 1]
[clojure.lang.Compiler eval "Compiler.java" 7176]
[clojure.lang.Compiler eval "Compiler.java" 7131]
[clojure.core$eval invokeStatic "core.clj" 3214]
[clojure.core$eval invoke "core.clj" 3210]
[clojure.main$repl$read_eval_print__9068$fn__9071 invoke "main.clj" 414]
[clojure.main$repl$read_eval_print__9068 invoke "main.clj" 414]
[clojure.main$repl$fn__9077 invoke "main.clj" 435]
[clojure.main$repl invokeStatic "main.clj" 435]
[clojure.main$repl doInvoke "main.clj" 345]Please let me know if there are any more gaps I can help fill in!
from libpython-clj.
ajpierce
commented on July 19, 2024
To follow up on :no-io-redirect?, we get the same behavior as before:
squishy-kitty.core=> (require '[libpython-clj.python :as py])
nil
squishy-kitty.core=> (py/initialize! :no-io-redirect? true)
Jan 10, 2020 12:54:30 PM clojure.tools.logging$eval1906$fn__1909 invoke
INFO: Executing python initialize!
Jan 10, 2020 12:54:31 PM clojure.tools.logging$eval1906$fn__1909 invoke
INFO: Startup info detected: {:python-home "/usr", :lib-version "3.7", :libname "python3.7m", :java-library-path-addendum "/usr/lib"}
Jan 10, 2020 12:54:31 PM clojure.tools.logging$eval1906$fn__1909 invoke
INFO: Reference thread starting
Jan 10, 2020 12:54:32 PM clojure.tools.logging$eval1906$fn__1909 invoke
INFO: Library python3.7m found at [:system "python3.7m"]
Execution error (IllegalArgumentException) at libpython-clj.jna.base/eval22533$fn$G (base.clj:20).
No implementation of method: :->py-object-ptr of protocol: #'libpython-clj.jna.base/PToPyObjectPtr found for class: nil
squishy-kitty.core=> *e
#error {
:cause "No implementation of method: :->py-object-ptr of protocol: #'libpython-clj.jna.base/PToPyObjectPtr found for class: nil"
:via
[{:type java.lang.IllegalArgumentException
:message "No implementation of method: :->py-object-ptr of protocol: #'libpython-clj.jna.base/PToPyObjectPtr found for class: nil"
:at [clojure.core$_cache_protocol_fn invokeStatic "core_deftype.clj" 583]}]
:trace
[[clojure.core$_cache_protocol_fn invokeStatic "core_deftype.clj" 583]
[clojure.core$_cache_protocol_fn invoke "core_deftype.clj" 575]
[libpython_clj.jna.base$eval22533$fn__22534$G__22524__22539 invoke "base.clj" 20]
[libpython_clj.jna.base$ensure_pyobj invokeStatic "base.clj" 50]
[libpython_clj.jna.base$ensure_pyobj invoke "base.clj" 48]
[libpython_clj.jna.concrete.unicode$PyUnicode_AsUTF8 invokeStatic "unicode.clj" 79]
[libpython_clj.jna.concrete.unicode$PyUnicode_AsUTF8 invoke "unicode.clj" 79]
[libpython_clj.jna.interpreter$get_type_name invokeStatic "interpreter.clj" 219]
[libpython_clj.jna.interpreter$get_type_name invoke "interpreter.clj" 216]
[clojure.lang.Var invoke "Var.java" 384]
[libpython_clj.python.interpreter$py_type_keyword$fn__24162 invoke "interpreter.clj" 340]
[clojure.lang.Atom swap "Atom.java" 37]
[clojure.core$swap_BANG_ invokeStatic "core.clj" 2352]
[clojure.core$swap_BANG_ invoke "core.clj" 2345]
[libpython_clj.python.interpreter$py_type_keyword invokeStatic "interpreter.clj" 333]
[libpython_clj.python.interpreter$py_type_keyword invoke "interpreter.clj" 327]
[libpython_clj.python.object$eval26422$fn__26423$fn__26424 invoke "object.clj" 231]
[libpython_clj.python.interpreter$with_gil_fn invokeStatic "interpreter.clj" 377]
[libpython_clj.python.interpreter$with_gil_fn invoke "interpreter.clj" 346]
[libpython_clj.python.object$eval26422$fn__26423 invoke "object.clj" 230]
[libpython_clj.python.protocols$eval24268$fn__24269$G__24259__24274 invoke "protocols.clj" 21]
[libpython_clj.python.protocols$python_type invokeStatic "protocols.clj" 30]
[libpython_clj.python.protocols$python_type invoke "protocols.clj" 26]
[libpython_clj.python.object$wrap_pyobject invokeStatic "object.clj" 136]
[libpython_clj.python.object$wrap_pyobject doInvoke "object.clj" 122]
[clojure.lang.RestFn invoke "RestFn.java" 410]
[libpython_clj.python.object$incref_wrap_pyobject$fn__26409 invoke "object.clj" 189]
[libpython_clj.python.interpreter$with_gil_fn invokeStatic "interpreter.clj" 377]
[libpython_clj.python.interpreter$with_gil_fn invoke "interpreter.clj" 346]
[libpython_clj.python.object$incref_wrap_pyobject invokeStatic "object.clj" 186]
[libpython_clj.python.object$incref_wrap_pyobject invoke "object.clj" 182]
[libpython_clj.python.interop$add_module$fn__26904 invoke "interop.clj" 87]
[libpython_clj.python.interpreter$with_gil_fn invokeStatic "interpreter.clj" 377]
[libpython_clj.python.interpreter$with_gil_fn invoke "interpreter.clj" 346]
[libpython_clj.python.interop$add_module invokeStatic "interop.clj" 85]
[libpython_clj.python.interop$add_module invoke "interop.clj" 83]
[libpython_clj.python.interop$register_bridge_type_BANG_$fn__26947 invoke "interop.clj" 281]
[libpython_clj.python.interpreter$with_gil_fn$fn__24167$fn__24168 invoke "interpreter.clj" 360]
[clojure.lang.AFn applyToHelper "AFn.java" 152]
[clojure.lang.AFn applyTo "AFn.java" 144]
[clojure.core$apply invokeStatic "core.clj" 665]
[clojure.core$with_bindings_STAR_ invokeStatic "core.clj" 1973]
[clojure.core$with_bindings_STAR_ doInvoke "core.clj" 1973]
[clojure.lang.RestFn invoke "RestFn.java" 425]
[libpython_clj.python.interpreter$with_gil_fn$fn__24167 invoke "interpreter.clj" 358]
[clojure.lang.AFn applyToHelper "AFn.java" 152]
[clojure.lang.AFn applyTo "AFn.java" 144]
[clojure.core$apply invokeStatic "core.clj" 665]
[clojure.core$with_bindings_STAR_ invokeStatic "core.clj" 1973]
[clojure.core$with_bindings_STAR_ doInvoke "core.clj" 1973]
[clojure.lang.RestFn invoke "RestFn.java" 425]
[libpython_clj.python.interpreter$with_gil_fn invokeStatic "interpreter.clj" 356]
[libpython_clj.python.interpreter$with_gil_fn invoke "interpreter.clj" 346]
[libpython_clj.python.interop$register_bridge_type_BANG_ invokeStatic "interop.clj" 280]
[libpython_clj.python.interop$register_bridge_type_BANG_ doInvoke "interop.clj" 276]
[clojure.lang.RestFn invoke "RestFn.java" 397]
[libpython_clj.python$initialize_BANG_ invokeStatic "python.clj" 244]
[libpython_clj.python$initialize_BANG_ doInvoke "python.clj" 229]
[clojure.lang.RestFn invoke "RestFn.java" 421]
[squishy_kitty.core$eval28099 invokeStatic "form-init7511438152933693024.clj" 1]
[squishy_kitty.core$eval28099 invoke "form-init7511438152933693024.clj" 1]
[clojure.lang.Compiler eval "Compiler.java" 7176]
[clojure.lang.Compiler eval "Compiler.java" 7131]
[clojure.core$eval invokeStatic "core.clj" 3214]
[clojure.core$eval invoke "core.clj" 3210]
[clojure.main$repl$read_eval_print__9068$fn__9071 invoke "main.clj" 414]
[clojure.main$repl$read_eval_print__9068 invoke "main.clj" 414]
[clojure.main$repl$fn__9077 invoke "main.clj" 435]
[clojure.main$repl invokeStatic "main.clj" 435]
[clojure.main$repl doInvoke "main.clj" 345]
[clojure.lang.RestFn invoke "RestFn.java" 1523]
[nrepl.middleware.interruptible_eval$evaluate invokeStatic "interruptible_eval.clj" 79]
[nrepl.middleware.interruptible_eval$evaluate invoke "interruptible_eval.clj" 55]
[nrepl.middleware.interruptible_eval$interruptible_eval$fn__939$fn__943 invoke "interruptible_eval.clj" 142]
[clojure.lang.AFn run "AFn.java" 22]
[nrepl.middleware.session$session_exec$main_loop__1040$fn__1044 invoke "session.clj" 171]
[nrepl.middleware.session$session_exec$main_loop__1040 invoke "session.clj" 170]
[clojure.lang.AFn run "AFn.java" 22]
[java.lang.Thread run "Thread.java" 748]]}from libpython-clj.
cnuernber
commented on July 19, 2024
That stack trace did help but I am stumped a bit. The module object created by calling 'add-module' has no __name__ attribute which seems to be a failure in the API.
There are a few questions I have at this point. What I am trying to do there is create a new module that holds the special python type we use to bridge things. Without that working almost none of the system will work.
It is odd that an added module wouldn't have a name attribute which I believe is the source of the issue.
I would like to play with this setup. Is there a way to get all this running in a VM or something like that so I can try it on my computer? Or, conversly, my buddy just got an embedded device and when I get back to Boulder I can grab it from him and try things out. For sure, there is a way to get this to work. It feels to me like there are just a few things that need to be tweaked for all of this to fly.
One way forward would be to hack libpython-clj to survive that possibility. If this code try/catches the error and just returns :unnamed-thing or something like that we could see what the next issue is. My guess is there will be 3 or for things like this before this setup works. If you are feeling bold, you can modify that file and locally install the library or modify it from within your repl and see if you can at least get the bridging type to register.
Great bug, this is interesting :-).
from libpython-clj.
owap
commented on July 19, 2024
Hi Chris, thanks again for the info!
The first thing I'm going to do is try to bypass the __name__ check. Then, probably in parallel, I'm going to see if I can get a VM to replicate what's happening on my hardware. If I can, I'll upload it to S3 and send you a link for downloading.
If I can't virtually reproduce what's happening, I'm going to go back to Microcenter and plunk down the ten whole dollars it'll cost me to buy another Pi Zero and see if I can replicate the exact environment I'm using (which, actually, the steps have been documented in this blog post and are continued in the next post.
If I'm successful physically but not virtually replicating the goings-on, I will reach out to you again for some shipping information and send you the hardware so you can play with it yourself :)
I'll report back later with developments!
from libpython-clj.
cnuernber
commented on July 19, 2024
That is great. One thing I noticed is that you are running in a 32 bit arm environment. While I tried to make everything just work on 32 bit environments, I haven't tested this and so almost certainly there are things hardcoded to live in the 64 bit world. I can work through at least getting things to work in a 32 bit vm and making sure all of that is sane which I think is a good parallel pathway to what you are up to.
from libpython-clj.
ajpierce
commented on July 19, 2024
I broke things!
First, let me say that my attempts to get a VM of Raspbian up and running have all resulted in failure. Part of me wonders if the issue lies in my host machine, but I haven't tried a base case of virtualizing something like Ubuntu or Debian yet. I may revisit that later.
On the "modify libpython-clj" front, I made two modifications locally:
diff --git a/src/libpython_clj/jna/interpreter.clj b/src/libpython_clj/jna/interpreter.clj
index 59dcabe..9270478 100644
--- a/src/libpython_clj/jna/interpreter.clj
+++ b/src/libpython_clj/jna/interpreter.clj
@@ -215,11 +215,14 @@
(defn get-type-name
[type-pyobj]
- (-> (pyobj/PyObject_GetAttrString type-pyobj "__name__")
- (pyuni/PyUnicode_AsUTF8)
- (jna/variable-byte-ptr->string)
- ->kebab-case
- keyword))
+ (println "@@@@ GETTING THE TYPE NAME @@@@")
+ (try
+ (-> (pyobj/PyObject_GetAttrString type-pyobj "__name__")
+ (pyuni/PyUnicode_AsUTF8)
+ (jna/variable-byte-ptr->string)
+ ->kebab-case
+ keyword)
+ (catch Throwable e :unnamed-thing)))
(defn lookup-type-symbols
diff --git a/src/libpython_clj/python/interop.clj b/src/libpython_clj/python/interop.clj
index 55713a3..7c8d8df 100644
--- a/src/libpython_clj/python/interop.clj
+++ b/src/libpython_clj/python/interop.clj
@@ -82,6 +82,7 @@
(defn add-module
[modname]
+ (println "IN MY SUPER MODIFIED add-module")
(with-gil
(-> (libpy/PyImport_AddModule modname)
(incref-wrap-pyobject))))When I attempted to start it up again, it actually segfaulted and crashed the REPL!
squishy-kitty.core=> (require '[libpython-clj.python :as py]) [968/1916]nil
squishy-kitty.core=> (py/initialize!)
Jan 12, 2020 1:00:16 PM clojure.tools.logging$eval1906$fn__1909 invoke
INFO: Executing python initialize!
Jan 12, 2020 1:00:18 PM clojure.tools.logging$eval1906$fn__1909 invoke
INFO: Startup info detected: {:python-home "/usr", :lib-version "3.7", :libname "python3.7m", :java-library-path-addendum "/usr/lib"}
Jan 12, 2020 1:00:18 PM clojure.tools.logging$eval1906$fn__1909 invoke
INFO: Reference thread starting
Jan 12, 2020 1:00:18 PM clojure.tools.logging$eval1906$fn__1909 invoke
INFO: Library python3.7m found at [:system "python3.7m"]
@@@@ GETTING THE TYPE NAME @@@@
@@@@ GETTING THE TYPE NAME @@@@
@@@@ GETTING THE TYPE NAME @@@@
@@@@ GETTING THE TYPE NAME @@@@
[...repeat ~40x]
@@@@ GETTING THE TYPE NAME @@@@
@@@@ GETTING THE TYPE NAME @@@@
IN MY SUPER MODIFIED add-module
@@@@ GETTING THE TYPE NAME @@@@
@@@@ GETTING THE TYPE NAME @@@@
#
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGSEGV (0xb) at pc=0xa3f1b11c, pid=9994, tid=0xa86fe460
#
# JRE version: OpenJDK Runtime Environment (8.0_152-b76) (build 1.8.0_152-b76)
# Java VM: OpenJDK Client VM (25.152-b76 mixed mode, Evaluation linux-aarch32 )
# Problematic frame:
# C [libpython3.7m.so+0x1ff11c] PyObject_GetAttrString+0x0
#
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unli$ited" before starting Java again
#
# An error report file with more information is saved as:
# /home/pi/squishy-kitty/hs_err_pid9994.log
#
# If you would like to submit a bug report, please visit:
# http://www.azulsystems.com/support/
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#
SocketException The transport's socket appears to have lost its connection to the nREPL server
nrepl.transport/bencode/fn--9182/fn--9183 (transport.clj:108)
Subprocess failed
nrepl.transport/bencode/fn--9182 (transport.clj:108)
nrepl.transport/fn-transport/fn--9150 (transport.clj:55)
clojure.core/binding-conveyor-fn/fn--5739 (core.clj:2030)
java.util.concurrent.FutureTask.run (FutureTask.java:266)
java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1149)
java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:624)
java.lang.Thread.run (Thread.java:748)
Bye for now!Here's the additional error information it spit out.
Now, it's very possible that I incorrectly caught the exception, or returned the wrong thing, but I am at least pleased that I got the whole checkout local library modification business sorted out 😄
Happy to do some asynchronous programming with you if there are changes you want me to stick into my local code and run. Also happy to hop on a hangout/zoom/screenshare and do some synchronous collaboration as well.
I'm going to head to Microcenter and pick up some more hardware to replicate it. Let me know which of the following you have laying around so I know what else to pick up:
- MicroUSB cable w/ 1Amp (at least) charger
- Mini HDMI to HDMI cable
- USB OTG (on to go) cable -- looks like micro USB to female USB-A
I can set up SSH for you, but you'll still need keyboard/monitor access when you get the hardware to connect it to your wifi before you can SSH to it.
Thanks again for the help! Looking forward to working with you!
from libpython-clj.
cnuernber
commented on July 19, 2024
I have none of those things lying around :-). Well, probably a micro-usb cable here or there. As I said, I think the first move is to get the entire tech stack passing unit tests (tech.jna, tech.datatype) on 32 bit platforms; I am not sure it will just yet.
In any case, this will be a push of mine coming up because I think it is interesting/useful. Looking forward to this pathway.
from libpython-clj.
obohrer
commented on July 19, 2024
I'm having a similar issue with my raspberrypi. I ran the tests for tech.jna & tech.datatype, they do work on the pi.
pi@raspberrypi:~/dev/tech.jna $ lein test
...
Ran 2 tests containing 2 assertions.
0 failures, 0 errors.
pi@raspberrypi:~/dev/tech.datatype $ lein test
...
Ran 133 tests containing 2341 assertions.
0 failures, 0 errors.
(Although for tech.datatype i had to bring down the number of elements in test/tech/v2/datatype/simple_ops_perf_test.clj to avoid an out of memory exception).
When running python/initialize! it crashes the JVM
The issue seems to be in jna.concrete.module:
C [libpython3.7m.so+0x2115cc]
Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
j com.sun.jna.Native.invokeInt(Lcom/sun/jna/Function;JI[Ljava/lang/Object;)I+0
j com.sun.jna.Function.invoke([Ljava/lang/Object;Ljava/lang/Class;ZI)Ljava/lang/Object;+211
j com.sun.jna.Function.invoke(Ljava/lang/reflect/Method;[Ljava/lang/Class;Ljava/lang/Class;[Ljava/lang/Object;Ljava/util/Map;)Ljava/lang/Object;+271
j com.sun.jna.Function.invoke(Ljava/lang/Class;[Ljava/lang/Object;Ljava/util/Map;)Ljava/lang/Object;+37
j com.sun.jna.Function.invoke(Ljava/lang/Class;[Ljava/lang/Object;)Ljava/lang/Object;+7
j libpython_clj.jna.concrete.module$PyModule_AddObject.invokeStatic(Ljava/lang/Object;Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object;+157
j libpython_clj.jna.concrete.module$PyModule_AddObject.invoke(Ljava/lang/Object;Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object;+9
j libpython_clj.python.interop$register_type_BANG_$fn__32211.invoke()Ljava/lang/Object;+712
J 597 c1 clojure.lang.AFn.applyToHelper(Lclojure/lang/IFn;Lclojure/lang/ISeq;)Ljava/lang/Object; (3238 bytes) @ 0x6bdc0850 [0x6bdbcae0+0x00003d70]
J 599 c1 clojure.lang.AFn.applyTo(Lclojure/lang/ISeq;)Ljava/lang/Object; (12 bytes) @ 0x6bdc723c [0x6bdc7200+0x0000003c]
J 993 c1 clojure.core$apply.invokeStatic(Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object; (21 bytes) @ 0x6be8ce4c [0x6be8cd90+0x000000bc]
j clojure.core$with_bindings_STAR_.invokeStatic(Ljava/lang/Object;Ljava/lang/Object;Lclojure/lang/ISeq;)Ljava/lang/Object;+13
j clojure.core$with_bindings_STAR_.doInvoke(Ljava/lang/Object;Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object;+12
J 2411 c1 clojure.lang.RestFn.invoke(Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object; (119 bytes) @ 0x6c09e008 [0x6c09df50+0x000000b8]
j libpython_clj.python.interop$register_type_BANG_.invokeStatic(Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object;+301
j libpython_clj.python.interop$register_type_BANG_.invoke(Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object;+6
j libpython_clj.python.interop$register_bridge_type_BANG_.invokeStatic(Lclojure/lang/ISeq;)Ljava/lang/Object;+524
j libpython_clj.python.interop$register_bridge_type_BANG_.doInvoke(Ljava/lang/Object;)Ljava/lang/Object;+6
j clojure.lang.RestFn.invoke()Ljava/lang/Object;+26
j libpython_clj.python$initialize_BANG_.invokeStatic(Lclojure/lang/ISeq;)Ljava/lang/Object;+205
j libpython_clj.python$initialize_BANG_.doInvoke(Ljava/lang/Object;)Ljava/lang/Object;+6
Where do you suggest to look at?
from libpython-clj.
ronnac
commented on July 19, 2024
I'm interested in a 32 bit compilation, too (armhf). I run Clojure on my Android smartphone via termux (spacemacs and lein-clojupyter are running fine).
from libpython-clj.
cnuernber
commented on July 19, 2024
OK, if we want to go here I think it will be instructive for all of us.
The python bindings assume 64 bit architectures in a few places. The first place we will have to get exactly right is the struct definitions, things like:
I should mention here the way I got these things exactly right the first time was by using c++, compiling little programs and outputting offsetof calculations for various members.
Then it is going through the JNA bindings and very carefully making sure that things that are 64 bits on our desktop machines are 32 bits in the bindings. There is a tech.v3.jna/size-t* set of definitions for doing something like this.
This is all theoretically possible of course but I think it is probably like 2-4 weeks of work of someone who cares if we want to go there we should branch the library and come up with a documented development flow that everyone can take part in.
At the same time we can probably simplify libpython-clj a bit; for example most of the stubbed out JNA interfaces aren't used. This will probably mean there is a 32bit release of the library and 32-bit specific java/clojure sections.
A similar translation would be needed to use libpython with a python library built with reference tracing as that changes the member variables on each python object.
from libpython-clj.
ronnac
commented on July 19, 2024
I was probably too quick. Graal is not available for ARM32. Is that a dependency?
from libpython-clj.
cnuernber
commented on July 19, 2024
clj-python/libpython-clj-2.00-beta-1 now supports 32 bit architectures via a ton of work to the ffi interface in dtype-next. If you have a moment, could you please test this out?
from libpython-clj.
ronnac
commented on July 19, 2024
Hi Chris,That is very good news! I will try it out. Please give me some time, I have a lot of work these days.I try do it in the coming week.
from libpython-clj.
Related Issues (20)
- Cannot run the example in the tutorial in Cider with JDK 17 HOT 8
- process hanging in embeded mode on exception HOT 2
- from-import does not support "string"
- newbie: I get: `dir already refers to: #'clojure.repl/dir in namespace: user` HOT 2
- Newbie: Lein Clojure Windows Anaconda VSCode Cava REPL Returns (NoClassDefFoundError) HOT 5
- Create Python Module then Import Module and Call Function Inside Clojure? HOT 4
- Fix ubuntu version in dockerfiles HOT 6
- Consider releasing returned values from instance fns
- JDK-20 support HOT 3
- builtins/eval throws 'frame does not exist' HOT 2
- Unable to require libraries from within docker using conda HOT 6
- Failed to find a valid python library on MS-Windows with the official python distributions
- Extend CI coverage to all the mainstream architectures
- require-python missing functions with annotation HOT 3
- Make polyglot development easier by allow require-python to be path-informed HOT 1
- Data transformation pipelines? HOT 1
- NSWindow main thread error when trying to setup Gymnasium (openapi gym) HOT 5
- JVM crashes when requiring `libpython-clj2.require` on 64bit Raspberry Pi 4 HOT 1
- error of unhashable type HOT 15
- Ways to reduce probability of memory leaks? HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from libpython-clj.