Fundamental memory leak in Modin about modin HOT 4 CLOSED

Hi @VladVin, thanks for posting this!

The problem in Modin 0.1.2 is not a memory leak, rather a result of the way that Ray + Arrow's Plasma store handle objects that are no longer referenced. We depend on both of these projects for the backend of Modin. I will explain the way that Ray and Arrow handle python objects put into memory and then explain how this will be fixed in 0.2 (and is partly fixed in current master).

The way the Ray interacts with the plasma store is that it will continue adding objects to the plasma store until a new object cannot be added. This includes objects that are no longer being referenced from the Python applications. Once it encounters this, it will evict objects in FIFO order. This can be particularly problematic for most workloads in Modin. Recently @robertnishihara added a feature to Ray that allows for manual freeing of objects from the plasma object store. We will be able to use it once Ray 0.6 is released.

In Modin 0.1.2 the copying/caching we do is excessive. That will add to the memory usage of each DataFrame, and in 0.1.2 there is no way to free them. We rewrote the backend in #70 and it no longer has the memory overhead you will see in 0.1.2. We also added handles to free the objects once they are no longer being referenced. This way, when Ray 0.6 is released we can just add the 2 or 3 lines of code needed. We expect this to effectively solve the high memory usage problem.

To some degree, there will always be some copying, this is unavoidable with an immutable store as the backend (Plasma). However, there is a lot we can do to reduce this so you aren't using 110GB of memory for your 3GB dataset.

cc @robertnishihara @pcmoritz (Feel free to correct me if I'm wrong)

from modin.

One minor comment is that the object store evicts objects in a least-recently-used order instead of FIFO.

from modin.

@devin-petersohn , @robertnishihara , thank you for the comments very much. Nice to hear that it will be fixed in the near releases

from modin.

Closing this. Feel free to reopen if the discussion should continue or if issue was not resolved.

from modin.