Multiprocessing is Experiencing Deadlocks in GraphFrame.filter about hatchet HOT 3 OPEN

@JBlaschke how large is the Graph that you're testing with? When Pandas serializes into any format (e.g., pickle, HDF5, SQL, etc.), it will serialize any cells with Object type data with pickle. This could become a problem with the index of the DataFrame because it contains nodes in the Graph. Under normal circumstances, each of these nodes will contain two lists: one of its parent nodes and one of its children nodes. So, when Pandas tries to pickle a node, it will recursively try to pickle each child and parent node. As a result, for large or highly connected graphs, it would be pretty easy to reach Python's (relatively small) recursion depth limit.

This is why, in developing #272 for saving GraphFrames to the filesystem, I disconnect the graph and encode that info with integers in a copy of the DataFrame.

@pearce8 @bhatele, we should probably discuss this during our next meeting.

from hatchet.

This library could be useful for fixing this issue: https://github.com/jmcarpenter2/swifter

It's a Pandas plugin that automatically decides if vectorization, Dask, or Pandas is faster and runs the apply function using that method. The syntax is pretty nice too. Instead of df.apply(func), Swifter's syntax is df.swifter.apply(func).

Only question is: do we want to add more dependencies?

from hatchet.

Actually, we probably don't want to use that package. It requires Pandas 1.0.0 or higher, and that version of Pandas doesn't support Python 2.7 (officially, it might work, but it's not a guarantee).

from hatchet.