[BUG] Dataloader doesnt release memory and memory growth about dataloader HOT 6 CLOSED

We can now create a context manager with dataloader (#78), so we can properly release the memory after each loop with:

train = nvt.Dataset(files, part_size="100MB")
for j in range(250):
    with Loader(train, batch_size=1024 * 64, shuffle=True, drop_last=True) as train_dl:
    for i, (inputs, labels) in enumerate(train_dl):
        # do stuff

instead of using dataloader.stop() which is easy to forget to do.

I confirmed that the memory is properly released by using a context manager in the original script.

from dataloader.

The potential solution is to call train_dl.stop() before del train, train_dl.

from dataloader.

@benfred , please check if there is a fix from @oliverholworthy and link it. Thank you

from dataloader.

@bschifferer did you test that solution and did it work?

from dataloader.

I need to call train_dl.stop() that all elements in the queue are deleted

from dataloader.

one hypothesis is we might need to implement __del__ to automatically clean itself up for the case where the dataloader doesn't get entirely consumed and the object goes out of scope. Or make it possible to use the loader as a context manager to handle this.

from dataloader.