How to shuffle selected partition after SplitDataset ? about torchnet HOT 4 CLOSED

Note that d = d:shuffle() is syntactic sugar for doing d = tnt.ShuffleDataset{dataset = d}. So of course select() is gone in d.

Can't you do e = d:shuffle() and get samples from e:iterator()?

from torchnet.

It works now, thanks 👍

local tnt = require 'torchnet'

local d = tnt.TableDataset{data = {1, 2, 3, 4, 5, 6, 7, 8, 9, 10}}

d = d:split{
  train = 0.7,
  val   = 0.3
}

d:select('val')
print('val')
for sample in d:iterator()() do
  print(sample)
end

d:select('train')
print('train')
for sample in d:iterator()() do
  print(sample)
end

trainset=d:shuffle()
print('train2')
for sample in trainset:iterator()() do
  print(sample)
end

d:select('val')
testset=d:shuffle()
print('val2')
for sample in testset:iterator()() do
  print(sample)
end

from torchnet.

Be careful with this though! Your call to d:select('val') also changes trainset because SplitDataset.select is not actually cloning. So both trainset and testset have the same underlying dataset d, with the same partition selected at any point in time.

We may have to consider changing SplitDataset to actually return a table of Datasets to prevent these kinds of bugs from occurring.

from torchnet.

Oh so it didn't solve the problem. That's what I thought in the first place. I really think returning a table of Dataset is a good idea.

I found this issue thinking about a possible BootstrapDataset which could split a Dataset in two partitions :

• the first one (train) would be made with a shuffle{remplacement=true} call,
• the second one (val) would be made from all the non picked samples from the first partition.

from torchnet.