Thread Safety about execnet HOT 8 CLOSED

Original comment by @RonnyPfannschmidt

after some investigation it seems like we need a utility function with a lock to have that one work properly

from execnet.

Original comment by @RonnyPfannschmidt

i created a proof of concept patch

please try https://bitbucket.org/RonnyPfannschmidt/execnet-patches/src/57cca2079529/linecache

after you report i'll commit it to the main repo

from execnet.

Original comment by Anonymous

Hi Ronny,

i looked at the patch and i guess it will work this way. I do understand that you want to always reload the module from disk when something changes but if i just want to execute the same module remotely lets say ten thousand times on different hosts it will result in reading the same module ten thousand times from disc. With your patch these ten thousand reads would be sequentially. Lets say the os and hard drive wouldnt cache HDD access. With 15ms hdd access speed it would take 150s to only load the module.

from execnet.

Original comment by @RonnyPfannschmidt

sadly i cant make linecache threadsafe,

however i can rely on disk-caches (if you load the same files thousands of times, its bound to end in cache)

a propper cache would be nice to have

could you give a more detailed example of your use-case,

we might be able to adapt things like execnet groups to do the file loading only once for large sets of gateways if it fits your use-case

from execnet.

Original comment by Anonymous

What iam doing right now with execnet:

{{{
I have n nodes.
For each node I create one group.
For each group I create one gateway.
}}}

Over each gateway I create x channels in different threads which are active in parallel with:
{{{
channel = gw.remote_exec(my_module) # this results in linecache activity
}}}

All channels load the same module (my_module). The remote code needs to be reloaded for a new channel
because its in a non recoverable state after execution. This means I cant ('without a lot of effort') let the channel stay open and send the next job.

Without looking at the remote_exec code a possibility could be something like this:

{{{
#!python

def remote_exec(self, ..., reload_module=True):
...

 if reload_module:
        linecache.updatecache(inspect.getsourcefile(source))

 ...

}}}

with this solution u wont break anyones production code if they decide to use a new version.

from execnet.

Original comment by @RonnyPfannschmidt

remote_exec cant have new args with special meaning

you should have one group with n nodes instead of n groups with 1 node

then we can do a group.remote_exec that will load once, send everywhere
the result will be a multichannel

from execnet.

Original comment by @RonnyPfannschmidt

btw, please drop by in the #pylib channel on the freenode irc network if possible

from execnet.

Original comment by @RonnyPfannschmidt

unresolved - feedback died down

from execnet.