Exception: namespace 'flambo.function' not found about flambo HOT 18 CLOSED

See the section on AOT in the README https://github.com/yieldbot/flambo#aot

from flambo.

The requiring namespaces are aot'd as per the readme.
This exception shows up when flambo.api refers in the flambo.function vars and seems to require flambo.function to be aot'd.

from flambo.

since flambo.api requires flambo.function, AOTing the all namespaces that require flambo.api should be all that's required. If you've already done this, perhaps you can make me a gist of your project.clj and I might be able to help further.

from flambo.

https://gist.github.com/alilee/565d0097b4c36edcbd26

I'm starting to have issues when compiling/re-constructing serialised functions with closures (referencing let or param bindings inside the f/fn). Could this be related?

Thanks very much for your help.

from flambo.

I think this could be related to some subtleties in how serialised-fn works - there isn't much guidance in the docs. This is what I'm having trouble with:

(let [x {:f identity :g parse/parse-int}]
  (-> sc 
      (f/parallelize [1 2 3 4 5])
      (f/map (f/fn [i] x))
      (f/take 1)))

If you take :g parse/parse-int out, then it works fine. When I leave it in it gets clojure.lang.Compiler$CompilerException: java.lang.RuntimeException: Unable to resolve symbol: x in this context, compiling:(NO_SOURCE_PATH:0:0) coming back from the worker.

parse is :aot.

from flambo.

parse/parse-int must also be a serializable-fn, otherwise it cannot be
serialized by your anonymous serializable-fn.

On Tuesday, December 16, 2014, Alister Lee [email protected] wrote:

I think this could be related to some subtleties in how serialised-fn
works - there isn't much guidance in the docs. This is what I'm having
trouble with:

(let [x {:f identity :g parse/parse-int}](-> sc
%28f/parallelize [1 2 3 4 5]%29
%28f/map %28f/fn [i] x%29%29
%28f/take 1%29))

If you take :g parse/parse-int out, then it works fine. When I leave it
in it gets clojure.lang.Compiler$CompilerException:
java.lang.RuntimeException: Unable to resolve symbol: x in this context,
compiling:(NO_SOURCE_PATH:0:0) coming back from the worker.

parse is :aot.

—
Reply to this email directly or view it on GitHub
#42 (comment).

http://about.me/soren

from flambo.

I get the same issue with this:

(let [x {:f identity :g (f/fn [s] (parse/parse-int s))}]
  (-> sc 
      (f/parallelize [1 2 3 4 5])
      (f/map (f/fn [i] x))
      (f/take 1)))

Is that what you were suggesting?

from flambo.

Hmmm. And same with this:

(let [x {:f identity :g (f/fn [s] (identity s))}]
  (-> sc 
      (f/parallelize [1 2 3 4 5])
      (f/map (f/fn [i] x))
      (f/take 1)))

from flambo.

try moving your let statement directly around the anonymous fn (let [x ...] (f/fn [i] x))

from flambo.

(use 'serializable.fn)

(deserialize 
  (serialize 
    (let [x + 
          c {:f identity :g (f/fn [s] (identity s))}
      (fn [a b] (:f c)))))
;; clojure.lang.Compiler$CompilerException: java.lang.RuntimeException: Unable to resolve symbol: c in this context, compiling:(...)

Is this the issue? (from serializable-fn):
TODO
Recurse into collections with the deep function serialization

from flambo.

No, that is an old TODO. Deep recursion works. I'll try your example when
I'm back at a computer.

On Tuesday, December 16, 2014, Alister Lee [email protected] wrote:

(use 'serializable.fn)

(deserialize
(serialize
(let [x +
c {:f identity :g (f/fn [s](identity s))}
(fn [a b](:f c)))))
;; clojure.lang.Compiler$CompilerException: java.lang.RuntimeException: Unable to resolve symbol: c in this context, compiling:(...)

Is this the issue? (from serializable-fn):
TODO https://github.com/yieldbot/serializable-fn#todo
Recurse into collections with the deep function serialization

—
Reply to this email directly or view it on GitHub
#42 (comment).

http://about.me/soren

from flambo.

Thanks a million. I think that is the issue though. Is your version ahead of yieldbot's?

(let [x  {:f identity :g (f/fn [s] (identity s))}   ;; x would break mapping fi but y works
      y  (f/fn [s] (parse/parse-int s))
      fi (f/fn [i] (y i))]
  (-> sc 
      (f/parallelize ["9" "2" "3" "4" "5"])
      (f/map fi)
      (f/take 1)))
;; [9]

from flambo.

So-- it seems like there is a bug in the deep recursion where elements of a map are not recursed into for serialization properly. it is the :f identity key-value pair causing the issue. kryo is trying to serialize an instance of clojure.lang.AFunction, which is can't do. I'll see if I can fix the bug.

from flambo.

Fantastic - thanks very much again for all your help. I will test!

from flambo.

Ok, so it's not really a bug per se. serializable.fn uses a library called carbonite, which is a library for using kryo to serialize clojure data structures. When serializable.fn serializes an fn and it's bindings, it simply calls kryo's serialize, which is turn checks the class and then in the case of a clojure map, calls carbonite here:

https://github.com/sritchie/carbonite/blob/master/src/clj/carbonite/serializer.clj#L62-L69

carbonite doesn't know how to serialize a standard clojure function, so it effectively just fails to do so.

Unfortunately there isn't a great fix, other than not using functions as keys or values in a map if you wish it to be serializable :/

from flambo.

Or just try the stuff from PR #27. I did take it into my fork (https://github.com/chrisbetz/flambo) and it works. I could check for you later today or you give it a try yourself.

Cheers,
Chris

Am 18.12.2014 um 05:22 schrieb Soren Macbeth [email protected]:

Ok, so it's not really a bug per se. serializable.fn uses a library called carbonite, which is a library for using kryo to serialize clojure data structures. When serializable.fn serializes an fn and it's bindings, it simply calls kryo's serialize, which is turn checks the class and then in the case of a clojure map, calls carbonite here:

https://github.com/sritchie/carbonite/blob/master/src/clj/carbonite/serializer.clj#L62-L69

carbonite doesn't know how to serialize a standard clojure function, so it effectively just fails to do so.

Unfortunately there isn't a great fix, other than not using functions as keys or values in a map if you wish it to be serializable :/

—
Reply to this email directly or view it on GitHub.

from flambo.

That works, but not in the repl. You'll have to AOT all your code, losing the ability to do ad-hoc or interactive spark stuff which may or may not matter to you.

@chrisbetz I would be quite curious to see if that works in your fork or not.

from flambo.

Hi,

took me some time. You might want to check commit 99c998c in my fork for the test, which is "green".

Happy holidays,

Chris

from flambo.