optimass / maximally_interfered_retrieval Goto Github PK

While running the following:

python er_main.py --method mir_replay --dataset split_cifar10 --mem_size 50

the following error occurs:

Traceback (most recent call last):
  File "er_main.py", line 178, in <module>
    buffer.add_reservoir(data, target, None, task)
  File "/raid/joseph/Maximally_Interfered_Retrieval/buffer.py", line 130, in add_reservoir
    assert idx_buffer.max() < self.bx.size(0), pdb.set_trace()
RuntimeError: invalid argument 1: cannot perform reduction function max on tensor with no elements because the operation does not have an identity at /opt/conda/conda-bld/pytorch_1579022060824/work/aten/src/THC/generic/THCTensorMathReduce.cu:85

Reason is because idx_buffer is coming as empty, Please see this image:

PyTorch version: 1.4.0

Hi. Thanks for sharing your code.
I tested ER-MIR with 100 memories per class on Cifar10 several times, but get highest average accuracy of 0.454 +/- 0.018, lower than the accuracy of 0.476 +/- 0.011 in the original paper.
I didn't change the code, and I ran the code as described in Scripts/ER_experiments.sh. Is there any configurations that should be modified to reproduce your results? Thank you very much!

Hi, did you run any experiments on MiniImagenet for either doing nothing (finetuning) or stationary data?
I ask because I am getting 8.4% accuracy for 1 epoch of stationary training (rises to 33.1% after 5 epochs), which is much lower than 15.4% for ER and 16.8% for MIR (1 epoch, 1 iteration, 10k total buffer size, running your code). I am wondering how this can be? And whether you observed anything similar.

Hi,
the license under which this code is released is unclear.
Would you mind adding a license file?
https://docs.github.com/en/github/building-a-strong-community/adding-a-license-to-a-repository

Thanks.

Hi @optimass, thank you for your interesting work.

Just to clarify the sizes. For e.g. python er_main.py --method mir_replay --dataset split_cifar10 --mem_size 20 --subsample 50 --samples_per_task -1 --n_runs 5 --disc_iters 1 --suffix 'ER_MIR' this seems to be the flow:

• Total buffer size: 20 * 10 classes = 200 samples (line)
• Randomly sample 50 from the 200, from classes other than the current batch's 2 classes (line)
• For MIR: get top buffer_batch_size=10 from the 50 samples, using MIR metric to rank (line). For plain ER: randomly get 10 from the 50.

Is this correct?
What is the purpose of sub-sampling 50 as opposed to ranking the top 10 from the full 200?

args.mem_size = args.mem_sizeargs.n_classes #convert from per class to total memory
or
args.buffer_size = args.mem_sizeargs.n_classes #convert from per class to total memory

Does disc_epochs make sense in er_main.py? Is the data being trained with only one epoch

Hi,

Thanks for the nice work! Can you please upload the Mini-ImageNet data that were used with this repo.

It expects to be present at @eugenium 's /home. Link

Or, can you please share where I could download it from. This would help me maintain consistency with the reported results.

Thanks!
Joseph

Hi, I found a possible bug in the test accuracy calculation. For each task, you are averaging the accuracy across mini batches as the task accuracy.

This calculation is wrong when the total number of data is not divisible by the mini batch size. Please consider changing to counting the total correctly predicted samples per task and then divide it by the total number of test data in that task.