mattja / distob Goto Github PK

distob works fine with ipyparallel 4 and earlier, but gives spurious messages with ipyparallel 5.0.x and breaks with ipyparallel >=5.1.x, working some of the time, and completely freezing with no error message at other times.

Have not yet investigated this to find whether this is a bug in ipyparallel or distob, or whether it results from a change to the AsyncResult interface in ipyparallel.
Until this issue is sorted out, temporarily require ipyparallel 4.x

Hello,

Sounds a good project.
Is there a way to pass / scatter a dictionnary of 500k keys ?
Dictionary is dictionnary of dictionnary... to handle complex data.

How the data are copied (because I have 5go and dont have to copy all....)

Currently caching relies on the id() of the proxy object as part of the key for storing cached results.
This is wrong, as the id is only guaranteed to be unique among objects existing at the same time.

Result: If a proxy object is deleted and later another proxy with the same interface happens to be created at the same address in memory (not unlikely) then the new proxy may return cached results that belong to the old proxy.

Solution: Instead of a global cache, have one cache instance per proxy, with only the total memory usage managed globally

Hello,

Find your project interresting.
Just wondering if there is a way to generate many instances of a class: c1, c2,...cn (all having different parameters) and send them on the clusters and do individual compute ( c1.compute() ) and get back the results into in the local computer ?

Thanks