"not enough memory" with small dataset when nthread>1 for ParallelDatasetIterator about torchnet HOT 4 CLOSED

So when you create a ParallelDatasetIterator, the dataset (in your case, the BatchDataset) gets serialized in order for it to be copied to the threads. For data that lives in Torch tensors or in tds objects, shared serialization is used; so this should not lead to high additional memory usage. For Lua objects (strings, tables, etc.), an actual copy is performed, which costs memory.

If you are using LuaJIT, the memory used for these Lua objects will count to LuaJIT's 1-2G memory limit depending on your the platform you're compiling to; see https://kvitajakub.github.io/2016/03/08/luajit-memory-limitations/

Your error message makes me think that is the memory limit you're hitting here. The best way to resolve this is to try and find out which Lua tables in your dataset take up most of the memory, and copy those into tds.Vec / tds.Hash objects.

from torchnet.

@lvdmaaten Thanks for explaining. I follow the usage of ParallelDatasetIterator in the mnist example of torchnet:

local function getIterator(mode)
return tnt.ParallelDatasetIterator{
nthread = 1,
init = function() require 'torchnet' end,
closure = function()
return tnt.BatchDataset{
batchsize = 128,
dataset = tnt.ListDataset(file_list_text,function(fp)
local data, gt = torch.load(fp)
return {input=data,target=gt}
end)
Each file listed in "file_list_text" is a torch serialization file of size 100KB-200KB.
When BatchDataset gets seriallized, I think it won't actually load these data files. I don't see what could cause the memory consumption here. I tried using tds.Hash instead of "{input=data,target=gt}", but the code won't run:
FATAL THREAD PANIC: (write) /root/torch/install/share/lua/5.1/torch/File.lua:141: Unwritable object at <?>.callback.closure.tds.C

from torchnet.

Add require 'tds' to the init closure to resolve those kinds of serialization issues. Note that ParallelDatasetIterator is a relatively thin wrapper on top of torch-threads, so it may be instructive to read up on how torch-threads works: https://github.com/torch/threads

from torchnet.

Adding require 'tds' to the init closure makes no difference.
The ParallelDatasetIterator usage in mnist example should be the standard way to work with it. I just can't get it to work. In my case, if the number of samples increases to 220,000, it won't even work when nthread=1. I guess I have to look at other ways, as you suggest. Thanks, Ivdmaaten.

from torchnet.