Comments (22)
cnuernber
commented on June 20, 2024
4
https://github.com/cnuernber/libpython-clj/blob/master/example/src/mxnet_autograd.clj
with is now included in python.clj. And tested in the unit tests :-). Thank you for this issue, it rocked :-)!!
from libpython-clj.
cnuernber
commented on June 20, 2024
3
Thumbs up on the Julia interest, btw :-).
from libpython-clj.
cnuernber
commented on June 20, 2024
Something like:
user> (defmacro with
[bind-vec & body]
(let [varnames (map first (partition 2 bind-vec))]
`(let [~@bind-vec]
(try
~@(->> varnames
(map (fn [var-sym]
`(py/call-attr ~var-sym "__enter__"))))
~@body
(finally
~@(->> varnames
(map (fn [var-sym]
`(py/call-attr ~var-sym "__exit__")))))))))
#'user/with
user> (with [f (py/call-attr io "open" "project.clj")]
(py/call-attr f "readlines"))
['(defproject cnuernber/libpython-clj "1.9-SNAPSHOT"\n', ' :description "libpython bindings to the techascent ecosystem"\n', ' :url "http://github.com/cnuernber/libpython-clj"\n', ' :license {:name "EPL-2.0"\n', ' :url "https://www.eclipse.org/legal/epl-2.0/"}\n', ' :dependencies [[org.clojure/clojure "1.10.1"]\n', ' [techascent/tech.datatype "4.46"]\n', ' [camel-snake-kebab "0.4.0"]]\n', ' :repl-options {:init-ns user}\n', ' :java-source-paths ["java"])\n']
user> (macroexpand '(with [f (py/call-attr io "open" "project.clj")]
(py/call-attr f "readlines")))
(let*
[f (py/call-attr io "open" "project.clj")]
(try
(libpython-clj.python/call-attr f "__enter__")
(py/call-attr f "readlines")
(finally (libpython-clj.python/call-attr f "__exit__"))))from libpython-clj.
chanshunli
commented on June 20, 2024
WowοΌVery Cool ! ! ! Thanks Chris
from libpython-clj.
chanshunli
commented on June 20, 2024
(defonce autograd (py-import "mxnet.autograd"))
(defonce nd (-a> mxnet nd))
(defmacro $ [py-import-lib attr-name & args]
`(if (= (get (System/getenv) "LEIN_TYPE") "FRONTEND")
nil
(call-attr
~py-import-lib ~(str attr-name) ~@args)))
;; (-a> ($ np ones [2 3]) shape) ;; => (2, 3)
(defmacro -a> [py-import-lib attr-name]
`(if (= (get (System/getenv) "LEIN_TYPE") "FRONTEND")
nil
(get-attr
~py-import-lib ~(str attr-name))))
;; x = nd.arange(4).reshape((4, 1))
(def x (-> nd
($ arange 4)
($ reshape [4 1])))
($ x attach_grad)
;; 2 * nd.dot(x.T, x)
(with [record ($ autograd record)]
(def y ($ nd multiply 2
($ nd dot (-a> x T) x)))
($ y backward))
from libpython-clj.
cnuernber
commented on June 20, 2024
Ahhh, the joys of native library development! Takes me back to my days as a much younger man...
Could you attach hs_err_pid12832.log? Seems odd that strlen would crash anything, maybe an error happened and the error reporting system is then causing a crash.
from libpython-clj.
chanshunli
commented on June 20, 2024
the python is here:
In [1]: from mxnet import autograd, nd
In [2]: x = nd.arange(4).reshape((4, 1))
In [3]: x.attach_grad()
In [4]: with autograd.record():
...: y = 2 * nd.dot(x.T, x)
...: y.backward()
...:
In [6]: x.grad
Out[6]:
[[ 0.]
[ 4.]
[ 8.]
[12.]]
<NDArray 4x1 @cpu(0)>
In [7]:
from libpython-clj.
chanshunli
commented on June 20, 2024
@cnuernber I have sent the log file to your mailbox [email protected].
Environment Variables:
CLASSPATH=/Users/clojure/.lein/self-installs/leiningen-2.9.1-standalone.jar
SHELL=/bin/zsh
Signal Handlers:
SIGSEGV: [libjvm.dylib+0x5b1141], sa_mask[0]=11111111011111110111111111111111, sa_flags=SA_ONSTACK|SA_RESTART|SA_SIGINFO
SIGBUS: [libjvm.dylib+0x5b1141], sa_mask[0]=11111111011111110111111111111111, sa_flags=SA_RESTART|SA_SIGINFO
SIGFPE: [libjvm.dylib+0x487bc4], sa_mask[0]=11111111011111110111111111111111, sa_flags=SA_RESTART|SA_SIGINFO
SIGPIPE: SIG_IGN, sa_mask[0]=00000000000000000000000000000000, sa_flags=SA_RESTART
SIGXFSZ: SIG_IGN, sa_mask[0]=00000000000000000000000000000000, sa_flags=none
SIGILL: [libjvm.dylib+0x487bc4], sa_mask[0]=11111111011111110111111111111111, sa_flags=SA_RESTART|SA_SIGINFO
SIGUSR1: SIG_DFL, sa_mask[0]=11011111011111100000000111000110, sa_flags=none
SIGUSR2: [libjvm.dylib+0x4876e2], sa_mask[0]=00100000000000000000000000000000, sa_flags=SA_RESTART|SA_SIGINFO
SIGHUP: [libjvm.dylib+0x485cb9], sa_mask[0]=11111111011111110111111111111111, sa_flags=SA_RESTART|SA_SIGINFO
SIGINT: [libjvm.dylib+0x485cb9], sa_mask[0]=11111111011111110111111111111111, sa_flags=SA_RESTART|SA_SIGINFO
SIGTERM: [libjvm.dylib+0x485cb9], sa_mask[0]=11111111011111110111111111111111, sa_flags=SA_RESTART|SA_SIGINFO
SIGQUIT: [libjvm.dylib+0x485cb9], sa_mask[0]=11111111011111110111111111111111, sa_flags=SA_RESTART|SA_SIGINFO
--------------- S Y S T E M ---------------
OS:Bsduname:Darwin 19.0.0 Darwin Kernel Version 19.0.0: Thu Oct 17 16:17:15 PDT 2019; root:xnu-6153.41.3~29/RELEASE_X86_64 x86_64
rlimit: STACK 9788k, CORE 0k, NPROC 1392, NOFILE 10240, AS infinity
load average:1.88 1.95 2.25
CPU:total 4 (2 cores per cpu, 2 threads per core) family 6 model 61 stepping 4, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3, sse4.1, sse4.2, popcnt, avx, avx2, aes, clmul, erms, 3dnowpref, lzcnt, ht, tsc, tscinvbit, bmi1, bmi2, adx
Memory: 4k page, physical 8388608k(218156k free)
/proc/meminfo:
vm_info: Java HotSpot(TM) 64-Bit Server VM (25.121-b13) for bsd-amd64 JRE (1.8.0_121-b13), built on Dec 12 2016 20:39:32 by "java_re" with gcc 4.2.1 (Based on Apple Inc. build 5658) (LLVM build 2336.11.00)
time: Fri Nov 15 10:04:40 2019
elapsed time: 348 seconds (0d 0h 5m 48s)from libpython-clj.
chanshunli
commented on June 20, 2024
@cnuernber Have you received my mail yet? Thanks Chris :-D
from libpython-clj.
cnuernber
commented on June 20, 2024
Yep, got it. There error happens when libpython-clj detects an error in the underlying code in a call-attr call. it tries to print it out and then bang. My guess is there is some context for mxnet that needs to be setup that isn't getting done.
Stack: [0x00007000066f7000,0x00007000067f7000], sp=0x00007000067f3e40, free space=1011k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C [libsystem_platform.dylib+0x1732] _platform_strlen+0x12
C [Python+0x790ac] unicode_fromformat_write_cstr+0x1f
C [Python+0x61010] PyUnicode_FromFormatV+0x69d
C [Python+0xbde11] PyErr_FormatV+0x29
C [Python+0xbd293] PyErr_Format+0x84
C [Python+0x46461] PyObject_GetAttr+0x99
C [Python+0x27430] PyFile_WriteObject+0x27
C [Python+0x27526] PyFile_WriteString+0x45
C [Python+0xd5d76] print_exception_recursive+0x53c
C [Python+0xd5797] PyErr_Display+0x6e
C [Python+0xdda51] sys_excepthook+0x4e
C [Python+0x19d4d] _PyMethodDef_RawFastCallDict+0xbf
C [Python+0x194aa] _PyCFunction_FastCallDict+0x2c
C [Python+0xd5462] PyErr_PrintEx+0x11b
C [jna5219763700594524346.tmp+0xde74] ffi_call_unix64+0x4c
C 0x00007000067f4ca8
Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
J 6781 com.sun.jna.Native.invokeVoid(Lcom/sun/jna/Function;JI[Ljava/lang/Object;)V (0 bytes) @ 0x0000000112b02729 [0x0000000112b026c0+0x69]
J 6690 C1 com.sun.jna.Function.invoke([Ljava/lang/Object;Ljava/lang/Class;ZI)Ljava/lang/Object; (769 bytes) @ 0x000000011418039c [0x000000011417b860+0x4b3c]
J 6694 C1 com.sun.jna.Function.invoke(Ljava/lang/reflect/Method;[Ljava/lang/Class;Ljava/lang/Class;[Ljava/lang/Object;Ljava/util/Map;)Ljava/lang/Object; (548 bytes) @ 0x00000001135ffc0c [0x00000001135fe2c0+0x194c]
J 6689 C1 com.sun.jna.Function.invoke(Ljava/lang/Class;[Ljava/lang/Object;Ljava/util/Map;)Ljava/lang/Object; (41 bytes) @ 0x000000011336adfc [0x000000011336a900+0x4fc]
J 6780 C1 com.sun.jna.Function.invoke([Ljava/lang/Object;)V (9 bytes) @ 0x0000000112a418b4 [0x0000000112a417c0+0xf4]
j libpython_clj.jna.concrete.err$PyErr_Print.invokeStatic()Ljava/lang/Object;+56
j libpython_clj.jna.concrete.err$PyErr_Print.invoke()Ljava/lang/Object;+0
J 6735 C1 clojure.lang.Var.invoke()Ljava/lang/Object; (10 bytes) @ 0x00000001133d9154 [0x00000001133d8ce0+0x474]
j libpython_clj.python.interpreter$check_error_str$fn__56102$fn__56103.invoke()Ljava/lang/Object;+11
J 4306 C2 clojure.lang.AFn.applyToHelper(Lclojure/lang/IFn;Lclojure/lang/ISeq;)Ljava/lang/Object; (3238 bytes) @ 0x00000001135c4f3c [0x00000001135c4380+0xbbc]
J 3884 C2 clojure.lang.AFn.applyTo(Lclojure/lang/ISeq;)Ljava/lang/Object; (12 bytes) @ 0x000000011349f014 [0x000000011349efe0+0x34]
J 6832 C1 clojure.core$with_bindings_STAR_.doInvoke(Ljava/lang/Object;Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object; (16 bytes) @ 0x0000000113332abc [0x0000000113332540+0x57c]
J 4882 C2 clojure.lang.RestFn.invoke(Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object; (119 bytes) @ 0x0000000113bd1ab0 [0x0000000113bd18e0+0x1d0]
j libpython_clj.python.interpreter$check_error_str$fn__56102.invoke()Ljava/lang/Object;+67
j libpython_clj.python.interpreter$with_gil_fn.invokeStatic(Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object;+465
J 6661 C1 libpython_clj.python.interpreter$with_gil_fn.invoke(Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object; (10 bytes) @ 0x000000011326f7bc [0x000000011326f740+0x7c]
j libpython_clj.python.interpreter$check_error_str.invokeStatic()Ljava/lang/Object;+17
j libpython_clj.python.interpreter$check_error_str.invoke()Ljava/lang/Object;+0
j libpython_clj.python.interpreter$check_error_throw.invokeStatic()Ljava/lang/Object;+9
j libpython_clj.python.interpreter$check_error_throw.invoke()Ljava/lang/Object;+0
j libpython_clj.python.object$wrap_pyobject.invokeStatic(Ljava/lang/Object;)Ljava/lang/Object;+9
j libpython_clj.python.object$wrap_pyobject.invoke(Ljava/lang/Object;)Ljava/lang/Object;+3
j libpython_clj.python.object$eval58298$fn__58299$fn__58300.invoke()Ljava/lang/Object;+204
j libpython_clj.python.interpreter$with_gil_fn$fn__56071.invoke()Ljava/lang/Object;+28
J 4306 C2 clojure.lang.AFn.applyToHelper(Lclojure/lang/IFn;Lclojure/lang/ISeq;)Ljava/lang/Object; (3238 bytes) @ 0x00000001135c4f3c [0x00000001135c4380+0xbbc]
J 3884 C2 clojure.lang.AFn.applyTo(Lclojure/lang/ISeq;)Ljava/lang/Object; (12 bytes) @ 0x000000011349f014 [0x000000011349efe0+0x34]
J 6832 C1 clojure.core$with_bindings_STAR_.doInvoke(Ljava/lang/Object;Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object; (16 bytes) @ 0x0000000113332abc [0x0000000113332540+0x57c]
J 4882 C2 clojure.lang.RestFn.invoke(Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object; (119 bytes) @ 0x0000000113bd1ab0 [0x0000000113bd18e0+0x1d0]
j libpython_clj.python.interpreter$with_gil_fn.invokeStatic(Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object;+110
J 6661 C1 libpython_clj.python.interpreter$with_gil_fn.invoke(Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object; (10 bytes) @ 0x000000011326f7bc [0x000000011326f740+0x7c]
j libpython_clj.python.object$eval58298$fn__58299.invoke(Ljava/lang/Object;Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object;+28
j libpython_clj.python.protocols$eval56462$fn__56463$G__56453__56472.invoke(Ljava/lang/Object;Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object;+60
j libpython_clj.python.bridge$generic_python_as_jvm$fn$reify__58812.do_call_fn(Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object;+50
j libpython_clj.python.protocols$call_attr.invokeStatic(Ljava/lang/Object;Ljava/lang/Object;Lclojure/lang/ISeq;)Ljava/lang/Object;+106
j libpython_clj.python.protocols$call_attr.doInvoke(Ljava/lang/Object;Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object;+12
J 4882 C2 clojure.lang.RestFn.invoke(Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object; (119 bytes) @ 0x0000000113bd1ab0 [0x0000000113bd18e0+0x1d0]
J 6955 C1 clojure.lang.Var.invoke(Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object; (24 bytes) @ 0x0000000112b7c6d4 [0x0000000112b7c1c0+0x514]
j luminus_shadow_txt2code.libpython.mxnet$eval65229.invokeStatic()Ljava/lang/Object;+256
j luminus_shadow_txt2code.libpython.mxnet$eval65229.invoke()Ljava/lang/Object;+0from libpython-clj.
chanshunli
commented on June 20, 2024
I don't know what to do after executing with autograd.record(): {code} , what did it do after the end.
Remove the with method, I don't know how to write in the mxnet.
from libpython-clj.
cnuernber
commented on June 20, 2024
One thing that could be going on is the with statement returns the last thing. The exit happens and then the repl attempts to print the last thing that was done within the context of the with. But the record has been exited by that time and this can cause a crash. In those cases I would return :ok as the last statement in the with to avoid that chance. I noticed in the python mxnet example they don't ever access variables defined within the record outside of the record.
from libpython-clj.
chanshunli
commented on June 20, 2024
How to use the gdb debug it, What happened inside with autograd.record(): {code} ?
like this:
/usr/local/bin/gdb -args /usr/local/Cellar/python/3.7.1/Frameworks/Python.framework/Versions/3.7/bin/python /Users/clojure/Desktop/hack_mxnet.py --core ./core from mxnet import autograd, nd
x = nd.arange(4).reshape((4, 1))
x.attach_grad()
print('before with autograd.record')
with autograd.record():
print('autograd.record ing ...')
y = 2 * nd.dot(x.T, x)
y.backward()
print('after with autograd.record')
print(x.grad)
from libpython-clj.
chanshunli
commented on June 20, 2024
@cnuernber Do you mean this? => I got this Error.
;; x = nd.arange(4).reshape((4, 1))
(def x (-> nd
($ arange 4)
($ reshape [4 1])))
($ x attach_grad)
;; 2 * nd.dot(x.T, x)
(with [record ($ autograd record)]
(do (let [y ($ nd multiply 2
($ nd dot (-a> x T) x))]
($ y backward))
"ok"))
from libpython-clj.
cnuernber
commented on June 20, 2024
Yes, that is what I meant.
The documentation link I posted above hints at what is going on. exit takes 3 additional positional arguments that I believe relate to the current in-flight exception. This will take more work to make solid, but you could modify the macro to call (py/call-attr ~argval "exit" nil nil nil) for now to see how that might work.
from libpython-clj.
chanshunli
commented on June 20, 2024
Although with {...} went wrong, x.grad is right. When I repl the with {...} , the program always crashes and exits.
from libpython-clj.
chanshunli
commented on June 20, 2024
Thanks @cnuernber Chris, It's work good π π π
(defmacro with
[bind-vec & body]
(let [varnames (map first (partition 2 bind-vec))]
`(let [~@bind-vec]
(try
~@(->> varnames
(map (fn [var-sym]
`(py/call-attr ~var-sym "__enter__"))))
~@body
(finally
~@(->> varnames
(map (fn [var-sym]
`(py/call-attr ~var-sym "__exit__" nil nil nil)))))))))
(comment
;; x = nd.arange(4).reshape((4, 1))
(def x (-> nd
($ arange 4)
($ reshape [4 1])))
($ x attach_grad)
;; 2 * nd.dot(x.T, x)
(with [record ($ autograd record)]
(let [y ($ nd multiply 2
($ nd dot (-a> x T) x))]
($ y backward))
)
(def dx (-a> x grad))
;; =>
;; [[ 0.]
;; [ 4.]
;; [ 8.]
;; [12.]]
;; <NDArray 4x1 @cpu(0)>
;;x = mx.nd.random.uniform(shape=(10,))
(def x1 (-> mxnet
(-a> nd)
(-a> random)
(call-attr-kw "uniform" [] {"shape" 10})))
($ x1 attach_grad)
(with [record ($ autograd record)]
(let [m ($ nd sigmoid x1)]
($ m backward)))
(def dx1 (-a> x1 grad)) ;; [0. 0. 0. 0. 0. 0. 0. 0. 0. 0.] => [0.23208144 0.22925895 0.2205717 0.21025445 0.22859976 0.20910253 0.23232493 0.2100039 0.23910952 0.22719035]
)
from libpython-clj.
cnuernber
commented on June 20, 2024
goooaaaallll!!!!!
That is awesome :-)!!
π π―
from libpython-clj.
chanshunli
commented on June 20, 2024
@cnuernber π₯ π― π Can you add with macro to the cnuernber/libpython-clj library?
from libpython-clj.
cnuernber
commented on June 20, 2024
I will eventually. It interacts with error handling so the 'correct' fix will involve some thinking.
from libpython-clj.
chanshunli
commented on June 20, 2024
@cnuernber Good Job π π π π― π― π― https://clojars.org/cnuernber/libpython-clj , I am updating to the 1.9 version right now.
from libpython-clj.
cnuernber
commented on June 20, 2024
π - best issue ever :-).
from libpython-clj.
Related Issues (20)
- Cannot run the example in the tutorial in Cider with JDK 17 HOT 8
- process hanging in embeded mode on exception HOT 2
- from-import does not support "string"
- newbie: I get: `dir already refers to: #'clojure.repl/dir in namespace: user` HOT 2
- Newbie: Lein Clojure Windows Anaconda VSCode Cava REPL Returns (NoClassDefFoundError) HOT 5
- Create Python Module then Import Module and Call Function Inside Clojure? HOT 4
- Fix ubuntu version in dockerfiles HOT 6
- Consider releasing returned values from instance fns
- JDK-20 support HOT 3
- builtins/eval throws 'frame does not exist' HOT 2
- Unable to require libraries from within docker using conda HOT 6
- Failed to find a valid python library on MS-Windows with the official python distributions
- Extend CI coverage to all the mainstream architectures
- require-python missing functions with annotation HOT 3
- Make polyglot development easier by allow require-python to be path-informed HOT 1
- Data transformation pipelines? HOT 1
- NSWindow main thread error when trying to setup Gymnasium (openapi gym) HOT 5
- JVM crashes when requiring `libpython-clj2.require` on 64bit Raspberry Pi 4 HOT 1
- error of unhashable type HOT 15
- Ways to reduce probability of memory leaks? HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
π Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. πππ
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google β€οΈ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from libpython-clj.