Hello,

I have been trying to execute the script “exp/exps_Numbers.sh” to reproduce the results for the MNIST-SVHN based Numbers dataset. However, I have been running into a few issues and would appreciate it if you could address them.

Based on the results I received, I noticed that there is a merged model in every task. For example, modelmerge_user1_mode_DS=NumbersDataset_nb_tasks_id=23f26144-1730-11ec-806c-506b4b3bff98.pth.tar be seen in all task folders. However, based on the diagram you drew in your paper, there should be single merged model after training is done on the server. Currently, I don’t see a single merged model. Rather I see a merged model for every task as seen below.

Is there a single merged model? If so, where is the single merged model located? How can I find it?

MERGED MODELS:
/scratch1/amala/DUA/results/train/NumbersDataset_nb_tasks/IMM/MLP_cl_100_100/gridsearch/demo/IMM_lr=0.001_bs=20_epochs=10_lmbL2trans=0.001_alpha=1/task_1/TASK_TRAINING/best_model.pth.tar
/scratch1/amala/DUA/results/train/NumbersDataset_nb_tasks/IMM/MLP_cl_100_100/gridsearch/demo/IMM_lr=0.001_bs=20_epochs=10_lmbL2trans=0.001_alpha=1/task_2/TASK_TRAINING/modelmerge_user1_mode_DS=NumbersDataset_nb_tasks_id=23f26144-1730-11ec-806c-506b4b3bff98.pth.tar
/scratch1/amala/DUA/results/train/NumbersDataset_nb_tasks/IMM/MLP_cl_100_100/gridsearch/demo/IMM_lr=0.001_bs=20_epochs=10_lmbL2trans=0.001_alpha=1/task_3/TASK_TRAINING/modelmerge_user1_mode_DS=NumbersDataset_nb_tasks_id=23f26144-1730-11ec-806c-506b4b3bff98.pth.tar
/scratch1/amala/DUA/results/train/NumbersDataset_nb_tasks/IMM/MLP_cl_100_100/gridsearch/demo/IMM_lr=0.001_bs=20_epochs=10_lmbL2trans=0.001_alpha=1/task_4/TASK_TRAINING/modelmerge_user1_mode_DS=NumbersDataset_nb_tasks_id=23f26144-1730-11ec-806c-506b4b3bff98.pth.tar
/scratch1/amala/DUA/results/train/NumbersDataset_nb_tasks/IMM/MLP_cl_100_100/gridsearch/demo/IMM_lr=0.001_bs=20_epochs=10_lmbL2trans=0.001_alpha=1/task_5/TASK_TRAINING/modelmerge_user1_mode_DS=NumbersDataset_nb_tasks_id=23f26144-1730-11ec-806c-506b4b3bff98.pth.tar

This is how my exps_Numbers.sh file looks like:

#!/bin/bash

grid="demo"

"# Numbers SETUP"
model="MLP_cl_100_100"
ds="numbers_nb" # Numbers dataset -> use MLP

############################################
"# TRAIN SERVER"
train_script="../train/main_train.py"
epochs="10"
bs="20"

"# FIM/MAS-IMM/LACL + Task Experts (Common training of server models)"
args="--method IMM --lr 0.001 --lmbL2trans 0.001 --bs $bs --epochs $epochs --model_name $model --gridsearch_name $grid --ds_name $ds"

"# MAS"

args="--method MAS --lr 0.001 --lmb 1 --bs $bs --epochs $epochs --model_name $model --gridsearch_name $grid --ds_name $ds"

"# EWC"

args="--method EWC --lr 0.001 --lmb 400 --bs $bs --epochs $epochs --model_name $model --gridsearch_name $grid --ds_name $ds"

"# LWF"

args="--method LWF --lr 0.001 --lmb 1 --bs $bs --epochs $epochs --model_name $model --gridsearch_name $grid --ds_name $ds"

"# JOINT"

args="--method JOINT --lr 0.001 --bs $bs --epochs $epochs --model_name $model --gridsearch_name $grid --ds_name $ds"

./run_wrapper.sh "$train_script $args"

############################################
"# ADAPT/TEST USERS"
test_script="../test/main_test.py"

"# FIM-IMM"
args="--method IMM --IMM_mode mode --bs 20 --model_name $model --gridsearch_name $grid --ds_name $ds"

"# MAS-IMM"

args="--method IMM --IMM_mode mode_MAS --bs 20 --model_name $model --gridsearch_name $grid --ds_name $ds"

"# MAS-LACL"

args="--method LA --LA_mode plain --bs 20 --model_name $model --gridsearch_name $grid --ds_name $ds"

"# FIM-LACL"

args="--method LA --LA_mode FIM --bs 20 --model_name $model --gridsearch_name $grid --ds_name $ds"

"# FT (No AdaBN applicable)"

args="--method FT --bs 20 --model_name $model --gridsearch_name $grid --ds_name $ds"

"# Same for other methods"
"# method="JOINT""
"#method="EWC""
"#method="LWF""
"# method="MAS""
"# args="--method $method --bs 20 --model_name $model --gridsearch_name $grid --ds_name $ds""

./run_wrapper.sh "$test_script $args"

After running the command, “bash exps_Numbers.sh”, I receive the following results:

targets:[[0, 1], [2, 3], [4, 5], [6, 7], [8, 9]]
Using downloaded and verified file: /scratch1/amala/DUA/data/datasets/SVHN/train_32x32.mat
Using downloaded and verified file: /scratch1/amala/DUA/data/datasets/SVHN/test_32x32.mat
*** TASK numbers=0,1 ***
task_targets=[0, 1]

• TASK numbers=0,1: training
• TASK numbers=0,1: SVHN user
• TASK numbers=0,1: MNIST user
  *** TASK numbers=2,3 ***
  task_targets=[2, 3]
• TASK numbers=2,3: training
• TASK numbers=2,3: SVHN user
• TASK numbers=2,3: MNIST user
  *** TASK numbers=4,5 ***
  task_targets=[4, 5]
• TASK numbers=4,5: training
• TASK numbers=4,5: SVHN user
• TASK numbers=4,5: MNIST user
  *** TASK numbers=6,7 ***
  task_targets=[6, 7]
• TASK numbers=6,7: training
• TASK numbers=6,7: SVHN user
• TASK numbers=6,7: MNIST user
  *** TASK numbers=8,9 ***
  task_targets=[8, 9]
• TASK numbers=8,9: training
• TASK numbers=8,9: SVHN user
• TASK numbers=8,9: MNIST user
  Custom MNIST+SVHN preprocessing finished
  targets:[[0, 1], [2, 3], [4, 5], [6, 7], [8, 9]]
  Using downloaded and verified file: /scratch1/amala/DUA/data/datasets/SVHN/train_32x32.mat
  Using downloaded and verified file: /scratch1/amala/DUA/data/datasets/SVHN/test_32x32.mat
  *** TASK numbers=0,1 ***
  task_targets=[0, 1]
• TASK numbers=0,1: training
• TASK numbers=0,1: SVHN user
• TASK numbers=0,1: MNIST user
  *** TASK numbers=2,3 ***
  task_targets=[2, 3]
• TASK numbers=2,3: training
• TASK numbers=2,3: SVHN user
• TASK numbers=2,3: MNIST user
  *** TASK numbers=4,5 ***
  task_targets=[4, 5]
• TASK numbers=4,5: training
• TASK numbers=4,5: SVHN user
• TASK numbers=4,5: MNIST user
  *** TASK numbers=6,7 ***
  task_targets=[6, 7]
• TASK numbers=6,7: training
• TASK numbers=6,7: SVHN user
• TASK numbers=6,7: MNIST user
  *** TASK numbers=8,9 ***
  task_targets=[8, 9]
• TASK numbers=8,9: training
• TASK numbers=8,9: SVHN user
• TASK numbers=8,9: MNIST user
  Custom MNIST+SVHN preprocessing finished
  RUNNING IMM(IMM) in 'train' mode
  STARTING FROM EXISTING MLP MODEL (name= MLP_cl_100_100 ) in /scratch1/amala/DUA/data/models/MLP_cl_100_100(in_dim=2352).pth.tar
  RUNNING IMM(IMM) in 'mode' mode
  <shared_utils.methods.IMM object at 0x14a25fe3d1c0>
  STARTING TESTING FOR METHOD: IMM_mode
  => For dir: /scratch1/amala/DUA/results/train/NumbersDataset_nb_tasks/IMM/MLP_cl_100_100/gridsearch/demo
  Testing for exp: IMM_lr=0.001_bs=20_epochs=10_lmbL2trans=0.001_alpha=1

USER 1

COLLECTED USER DATASETS:
/scratch1/amala/DUA/data/datasets/MNIST+SVHN/user_testsubset_SubsetSVHN_targets[0, 1].pth
/scratch1/amala/DUA/data/datasets/MNIST+SVHN/user_testsubset_SubsetSVHN_targets[2, 3].pth
/scratch1/amala/DUA/data/datasets/MNIST+SVHN/user_testsubset_SubsetSVHN_targets[4, 5].pth
/scratch1/amala/DUA/data/datasets/MNIST+SVHN/user_testsubset_SubsetSVHN_targets[6, 7].pth
/scratch1/amala/DUA/data/datasets/MNIST+SVHN/user_testsubset_SubsetSVHN_targets[8, 9].pth

COLLECTED USER MODELS:
/scratch1/amala/DUA/results/train/NumbersDataset_nb_tasks/IMM/MLP_cl_100_100/gridsearch/demo/IMM_lr=0.001_bs=20_epochs=10_lmbL2trans=0.001_alpha=1/task_1/TASK_TRAINING/best_model.pth.tar
/scratch1/amala/DUA/results/train/NumbersDataset_nb_tasks/IMM/MLP_cl_100_100/gridsearch/demo/IMM_lr=0.001_bs=20_epochs=10_lmbL2trans=0.001_alpha=1/task_2/TASK_TRAINING/best_model.pth.tar
/scratch1/amala/DUA/results/train/NumbersDataset_nb_tasks/IMM/MLP_cl_100_100/gridsearch/demo/IMM_lr=0.001_bs=20_epochs=10_lmbL2trans=0.001_alpha=1/task_3/TASK_TRAINING/best_model.pth.tar
/scratch1/amala/DUA/results/train/NumbersDataset_nb_tasks/IMM/MLP_cl_100_100/gridsearch/demo/IMM_lr=0.001_bs=20_epochs=10_lmbL2trans=0.001_alpha=1/task_4/TASK_TRAINING/best_model.pth.tar
/scratch1/amala/DUA/results/train/NumbersDataset_nb_tasks/IMM/MLP_cl_100_100/gridsearch/demo/IMM_lr=0.001_bs=20_epochs=10_lmbL2trans=0.001_alpha=1/task_5/TASK_TRAINING/best_model.pth.tar
**************************************** ITERATION 1 ****************************************

IMM preprocessing: 'mode' mode

DATASETS FOR MERGING (IWS, task_agnostic=False):
/scratch1/amala/DUA/data/datasets/MNIST+SVHN/tr_trainsubset_SubsetMNIST+SVHN_targetsnumbers=0,1.pth
/scratch1/amala/DUA/data/datasets/MNIST+SVHN/tr_trainsubset_SubsetMNIST+SVHN_targetsnumbers=2,3.pth
/scratch1/amala/DUA/data/datasets/MNIST+SVHN/tr_trainsubset_SubsetMNIST+SVHN_targetsnumbers=4,5.pth
/scratch1/amala/DUA/data/datasets/MNIST+SVHN/tr_trainsubset_SubsetMNIST+SVHN_targetsnumbers=6,7.pth
/scratch1/amala/DUA/data/datasets/MNIST+SVHN/tr_trainsubset_SubsetMNIST+SVHN_targetsnumbers=8,9.pth

MODELS FOR MERGING:
/scratch1/amala/DUA/results/train/NumbersDataset_nb_tasks/IMM/MLP_cl_100_100/gridsearch/demo/IMM_lr=0.001_bs=20_epochs=10_lmbL2trans=0.001_alpha=1/task_1/TASK_TRAINING/best_model.pth.tar
/scratch1/amala/DUA/results/train/NumbersDataset_nb_tasks/IMM/MLP_cl_100_100/gridsearch/demo/IMM_lr=0.001_bs=20_epochs=10_lmbL2trans=0.001_alpha=1/task_2/TASK_TRAINING/best_model.pth.tar
/scratch1/amala/DUA/results/train/NumbersDataset_nb_tasks/IMM/MLP_cl_100_100/gridsearch/demo/IMM_lr=0.001_bs=20_epochs=10_lmbL2trans=0.001_alpha=1/task_3/TASK_TRAINING/best_model.pth.tar
/scratch1/amala/DUA/results/train/NumbersDataset_nb_tasks/IMM/MLP_cl_100_100/gridsearch/demo/IMM_lr=0.001_bs=20_epochs=10_lmbL2trans=0.001_alpha=1/task_4/TASK_TRAINING/best_model.pth.tar
/scratch1/amala/DUA/results/train/NumbersDataset_nb_tasks/IMM/MLP_cl_100_100/gridsearch/demo/IMM_lr=0.001_bs=20_epochs=10_lmbL2trans=0.001_alpha=1/task_5/TASK_TRAINING/best_model.pth.tar
removed reg_params from loaded model
removed reg_params from loaded model
removed reg_params from loaded model
removed reg_params from loaded model
MODELS TO MERGE:
/scratch1/amala/DUA/results/train/NumbersDataset_nb_tasks/IMM/MLP_cl_100_100/gridsearch/demo/IMM_lr=0.001_bs=20_epochs=10_lmbL2trans=0.001_alpha=1/task_1/TASK_TRAINING/best_model.pth.tar
/scratch1/amala/DUA/results/train/NumbersDataset_nb_tasks/IMM/MLP_cl_100_100/gridsearch/demo/IMM_lr=0.001_bs=20_epochs=10_lmbL2trans=0.001_alpha=1/task_2/TASK_TRAINING/best_model.pth.tar
/scratch1/amala/DUA/results/train/NumbersDataset_nb_tasks/IMM/MLP_cl_100_100/gridsearch/demo/IMM_lr=0.001_bs=20_epochs=10_lmbL2trans=0.001_alpha=1/task_3/TASK_TRAINING/best_model.pth.tar
/scratch1/amala/DUA/results/train/NumbersDataset_nb_tasks/IMM/MLP_cl_100_100/gridsearch/demo/IMM_lr=0.001_bs=20_epochs=10_lmbL2trans=0.001_alpha=1/task_4/TASK_TRAINING/best_model.pth.tar
/scratch1/amala/DUA/results/train/NumbersDataset_nb_tasks/IMM/MLP_cl_100_100/gridsearch/demo/IMM_lr=0.001_bs=20_epochs=10_lmbL2trans=0.001_alpha=1/task_5/TASK_TRAINING/best_model.pth.tar
LOADED 5 MODELS in MEMORY
IMM PREPROCESSING: Mode mode, overwrite=False
*** ESTIMATING IWS on MODEL of TASK 1

• IW: MODEL of TASK 1, on DS of TASK=1
  Calculating precision matrix
  *** ESTIMATING IWS on MODEL of TASK 2
• IW: MODEL of TASK 2, on DS of TASK=2
  Calculating precision matrix
  *** ESTIMATING IWS on MODEL of TASK 3
• IW: MODEL of TASK 3, on DS of TASK=3
  Calculating precision matrix
  *** ESTIMATING IWS on MODEL of TASK 4
• IW: MODEL of TASK 4, on DS of TASK=4
  Calculating precision matrix
  *** ESTIMATING IWS on MODEL of TASK 5
• IW: MODEL of TASK 5, on DS of TASK=5
  Calculating precision matrix
  MERGE MODE=[MEAN=False, IW=True] Merging models for TASK 2
  NOT MERGING PARAM classifier.4.weight, as it is a head param name
  NOT MERGING PARAM classifier.4.bias, as it is a head param name
  => SAVED MERGED MODEL: /scratch1/amala/DUA/results/train/NumbersDataset_nb_tasks/IMM/MLP_cl_100_100/gridsearch/demo/IMM_lr=0.001_bs=20_epochs=10_lmbL2trans=0.001_alpha=1/task_2/TASK_TRAINING/modelmerge_user1_mode_DS=NumbersDataset_nb_tasks_id=23f26144-1730-11ec-806c-506b4b3bff98.pth.tar
  MERGE MODE=[MEAN=False, IW=True] Merging models for TASK 3
  NOT MERGING PARAM classifier.4.weight, as it is a head param name
  NOT MERGING PARAM classifier.4.bias, as it is a head param name
  => SAVED MERGED MODEL: /scratch1/amala/DUA/results/train/NumbersDataset_nb_tasks/IMM/MLP_cl_100_100/gridsearch/demo/IMM_lr=0.001_bs=20_epochs=10_lmbL2trans=0.001_alpha=1/task_3/TASK_TRAINING/modelmerge_user1_mode_DS=NumbersDataset_nb_tasks_id=23f26144-1730-11ec-806c-506b4b3bff98.pth.tar
  MERGE MODE=[MEAN=False, IW=True] Merging models for TASK 4
  NOT MERGING PARAM classifier.4.weight, as it is a head param name
  NOT MERGING PARAM classifier.4.bias, as it is a head param name
  => SAVED MERGED MODEL: /scratch1/amala/DUA/results/train/NumbersDataset_nb_tasks/IMM/MLP_cl_100_100/gridsearch/demo/IMM_lr=0.001_bs=20_epochs=10_lmbL2trans=0.001_alpha=1/task_4/TASK_TRAINING/modelmerge_user1_mode_DS=NumbersDataset_nb_tasks_id=23f26144-1730-11ec-806c-506b4b3bff98.pth.tar
  MERGE MODE=[MEAN=False, IW=True] Merging models for TASK 5
  NOT MERGING PARAM classifier.4.weight, as it is a head param name
  NOT MERGING PARAM classifier.4.bias, as it is a head param name
  => SAVED MERGED MODEL: /scratch1/amala/DUA/results/train/NumbersDataset_nb_tasks/IMM/MLP_cl_100_100/gridsearch/demo/IMM_lr=0.001_bs=20_epochs=10_lmbL2trans=0.001_alpha=1/task_5/TASK_TRAINING/modelmerge_user1_mode_DS=NumbersDataset_nb_tasks_id=23f26144-1730-11ec-806c-506b4b3bff98.pth.tar
  MERGED MODELS:
  /scratch1/amala/DUA/results/train/NumbersDataset_nb_tasks/IMM/MLP_cl_100_100/gridsearch/demo/IMM_lr=0.001_bs=20_epochs=10_lmbL2trans=0.001_alpha=1/task_1/TASK_TRAINING/best_model.pth.tar
  /scratch1/amala/DUA/results/train/NumbersDataset_nb_tasks/IMM/MLP_cl_100_100/gridsearch/demo/IMM_lr=0.001_bs=20_epochs=10_lmbL2trans=0.001_alpha=1/task_2/TASK_TRAINING/modelmerge_user1_mode_DS=NumbersDataset_nb_tasks_id=23f26144-1730-11ec-806c-506b4b3bff98.pth.tar
  /scratch1/amala/DUA/results/train/NumbersDataset_nb_tasks/IMM/MLP_cl_100_100/gridsearch/demo/IMM_lr=0.001_bs=20_epochs=10_lmbL2trans=0.001_alpha=1/task_3/TASK_TRAINING/modelmerge_user1_mode_DS=NumbersDataset_nb_tasks_id=23f26144-1730-11ec-806c-506b4b3bff98.pth.tar
  /scratch1/amala/DUA/results/train/NumbersDataset_nb_tasks/IMM/MLP_cl_100_100/gridsearch/demo/IMM_lr=0.001_bs=20_epochs=10_lmbL2trans=0.001_alpha=1/task_4/TASK_TRAINING/modelmerge_user1_mode_DS=NumbersDataset_nb_tasks_id=23f26144-1730-11ec-806c-506b4b3bff98.pth.tar
  /scratch1/amala/DUA/results/train/NumbersDataset_nb_tasks/IMM/MLP_cl_100_100/gridsearch/demo/IMM_lr=0.001_bs=20_epochs=10_lmbL2trans=0.001_alpha=1/task_5/TASK_TRAINING/modelmerge_user1_mode_DS=NumbersDataset_nb_tasks_id=23f26144-1730-11ec-806c-506b4b3bff98.pth.tar

AFTER PREPROCESSING USER DATASETS:
/scratch1/amala/DUA/data/datasets/MNIST+SVHN/user_testsubset_SubsetSVHN_targets[0, 1].pth
/scratch1/amala/DUA/data/datasets/MNIST+SVHN/user_testsubset_SubsetSVHN_targets[2, 3].pth
/scratch1/amala/DUA/data/datasets/MNIST+SVHN/user_testsubset_SubsetSVHN_targets[4, 5].pth
/scratch1/amala/DUA/data/datasets/MNIST+SVHN/user_testsubset_SubsetSVHN_targets[6, 7].pth
/scratch1/amala/DUA/data/datasets/MNIST+SVHN/user_testsubset_SubsetSVHN_targets[8, 9].pth

AFTER PREPROCESSING USER MODELS:
/scratch1/amala/DUA/results/train/NumbersDataset_nb_tasks/IMM/MLP_cl_100_100/gridsearch/demo/IMM_lr=0.001_bs=20_epochs=10_lmbL2trans=0.001_alpha=1/task_1/TASK_TRAINING/best_model.pth.tar
/scratch1/amala/DUA/results/train/NumbersDataset_nb_tasks/IMM/MLP_cl_100_100/gridsearch/demo/IMM_lr=0.001_bs=20_epochs=10_lmbL2trans=0.001_alpha=1/task_2/TASK_TRAINING/modelmerge_user1_mode_DS=NumbersDataset_nb_tasks_id=23f26144-1730-11ec-806c-506b4b3bff98.pth.tar
/scratch1/amala/DUA/results/train/NumbersDataset_nb_tasks/IMM/MLP_cl_100_100/gridsearch/demo/IMM_lr=0.001_bs=20_epochs=10_lmbL2trans=0.001_alpha=1/task_3/TASK_TRAINING/modelmerge_user1_mode_DS=NumbersDataset_nb_tasks_id=23f26144-1730-11ec-806c-506b4b3bff98.pth.tar
/scratch1/amala/DUA/results/train/NumbersDataset_nb_tasks/IMM/MLP_cl_100_100/gridsearch/demo/IMM_lr=0.001_bs=20_epochs=10_lmbL2trans=0.001_alpha=1/task_4/TASK_TRAINING/modelmerge_user1_mode_DS=NumbersDataset_nb_tasks_id=23f26144-1730-11ec-806c-506b4b3bff98.pth.tar
/scratch1/amala/DUA/results/train/NumbersDataset_nb_tasks/IMM/MLP_cl_100_100/gridsearch/demo/IMM_lr=0.001_bs=20_epochs=10_lmbL2trans=0.001_alpha=1/task_5/TASK_TRAINING/modelmerge_user1_mode_DS=NumbersDataset_nb_tasks_id=23f26144-1730-11ec-806c-506b4b3bff98.pth.tar
Testing on 5 task datasets
TESTING ON TASK 1
Testing Dataset = /scratch1/amala/DUA/data/datasets/MNIST+SVHN/user_testsubset_SubsetSVHN_targets[0, 1].pth
=> Testing model trained up to and including TASK 1
Testing on model = /scratch1/amala/DUA/results/train/NumbersDataset_nb_tasks/IMM/MLP_cl_100_100/gridsearch/demo/IMM_lr=0.001_bs=20_epochs=10_lmbL2trans=0.001_alpha=1/task_1/TASK_TRAINING/best_model.pth.tar
EVAL on prev heads: /scratch1/amala/DUA/results/train/NumbersDataset_nb_tasks/IMM/MLP_cl_100_100/gridsearch/demo/IMM_lr=0.001_bs=20_epochs=10_lmbL2trans=0.001_alpha=1/task_1/TASK_TRAINING/best_model.pth.tar
Overall Accuracy: 94.9137679041216
=> Testing model trained up to and including TASK 2
Testing on model = /scratch1/amala/DUA/results/train/NumbersDataset_nb_tasks/IMM/MLP_cl_100_100/gridsearch/demo/IMM_lr=0.001_bs=20_epochs=10_lmbL2trans=0.001_alpha=1/task_2/TASK_TRAINING/modelmerge_user1_mode_DS=NumbersDataset_nb_tasks_id=23f26144-1730-11ec-806c-506b4b3bff98.pth.tar
EVAL on prev heads: /scratch1/amala/DUA/results/train/NumbersDataset_nb_tasks/IMM/MLP_cl_100_100/gridsearch/demo/IMM_lr=0.001_bs=20_epochs=10_lmbL2trans=0.001_alpha=1/task_1/TASK_TRAINING/best_model.pth.tar
Overall Accuracy: 89.09675533469746
=> Testing model trained up to and including TASK 3
Testing on model = /scratch1/amala/DUA/results/train/NumbersDataset_nb_tasks/IMM/MLP_cl_100_100/gridsearch/demo/IMM_lr=0.001_bs=20_epochs=10_lmbL2trans=0.001_alpha=1/task_3/TASK_TRAINING/modelmerge_user1_mode_DS=NumbersDataset_nb_tasks_id=23f26144-1730-11ec-806c-506b4b3bff98.pth.tar
EVAL on prev heads: /scratch1/amala/DUA/results/train/NumbersDataset_nb_tasks/IMM/MLP_cl_100_100/gridsearch/demo/IMM_lr=0.001_bs=20_epochs=10_lmbL2trans=0.001_alpha=1/task_1/TASK_TRAINING/best_model.pth.tar
Overall Accuracy: 87.0798012277112
=> Testing model trained up to and including TASK 4
Testing on model = /scratch1/amala/DUA/results/train/NumbersDataset_nb_tasks/IMM/MLP_cl_100_100/gridsearch/demo/IMM_lr=0.001_bs=20_epochs=10_lmbL2trans=0.001_alpha=1/task_4/TASK_TRAINING/modelmerge_user1_mode_DS=NumbersDataset_nb_tasks_id=23f26144-1730-11ec-806c-506b4b3bff98.pth.tar
EVAL on prev heads: /scratch1/amala/DUA/results/train/NumbersDataset_nb_tasks/IMM/MLP_cl_100_100/gridsearch/demo/IMM_lr=0.001_bs=20_epochs=10_lmbL2trans=0.001_alpha=1/task_1/TASK_TRAINING/best_model.pth.tar
Overall Accuracy: 85.8228588132125
=> Testing model trained up to and including TASK 5
Testing on model = /scratch1/amala/DUA/results/train/NumbersDataset_nb_tasks/IMM/MLP_cl_100_100/gridsearch/demo/IMM_lr=0.001_bs=20_epochs=10_lmbL2trans=0.001_alpha=1/task_5/TASK_TRAINING/modelmerge_user1_mode_DS=NumbersDataset_nb_tasks_id=23f26144-1730-11ec-806c-506b4b3bff98.pth.tar
EVAL on prev heads: /scratch1/amala/DUA/results/train/NumbersDataset_nb_tasks/IMM/MLP_cl_100_100/gridsearch/demo/IMM_lr=0.001_bs=20_epochs=10_lmbL2trans=0.001_alpha=1/task_1/TASK_TRAINING/best_model.pth.tar
Overall Accuracy: 85.09207833966677
TESTING ON TASK 2
Testing Dataset = /scratch1/amala/DUA/data/datasets/MNIST+SVHN/user_testsubset_SubsetSVHN_targets[2, 3].pth
=> Testing model trained up to and including TASK 2
Testing on model = /scratch1/amala/DUA/results/train/NumbersDataset_nb_tasks/IMM/MLP_cl_100_100/gridsearch/demo/IMM_lr=0.001_bs=20_epochs=10_lmbL2trans=0.001_alpha=1/task_2/TASK_TRAINING/modelmerge_user1_mode_DS=NumbersDataset_nb_tasks_id=23f26144-1730-11ec-806c-506b4b3bff98.pth.tar
EVAL on prev heads: /scratch1/amala/DUA/results/train/NumbersDataset_nb_tasks/IMM/MLP_cl_100_100/gridsearch/demo/IMM_lr=0.001_bs=20_epochs=10_lmbL2trans=0.001_alpha=1/task_2/TASK_TRAINING/modelmerge_user1_mode_DS=NumbersDataset_nb_tasks_id=23f26144-1730-11ec-806c-506b4b3bff98.pth.tar
Overall Accuracy: 89.55903271692745
=> Testing model trained up to and including TASK 3
Testing on model = /scratch1/amala/DUA/results/train/NumbersDataset_nb_tasks/IMM/MLP_cl_100_100/gridsearch/demo/IMM_lr=0.001_bs=20_epochs=10_lmbL2trans=0.001_alpha=1/task_3/TASK_TRAINING/modelmerge_user1_mode_DS=NumbersDataset_nb_tasks_id=23f26144-1730-11ec-806c-506b4b3bff98.pth.tar
EVAL on prev heads: /scratch1/amala/DUA/results/train/NumbersDataset_nb_tasks/IMM/MLP_cl_100_100/gridsearch/demo/IMM_lr=0.001_bs=20_epochs=10_lmbL2trans=0.001_alpha=1/task_2/TASK_TRAINING/modelmerge_user1_mode_DS=NumbersDataset_nb_tasks_id=23f26144-1730-11ec-806c-506b4b3bff98.pth.tar
Overall Accuracy: 90.49786628733997
=> Testing model trained up to and including TASK 4
Testing on model = /scratch1/amala/DUA/results/train/NumbersDataset_nb_tasks/IMM/MLP_cl_100_100/gridsearch/demo/IMM_lr=0.001_bs=20_epochs=10_lmbL2trans=0.001_alpha=1/task_4/TASK_TRAINING/modelmerge_user1_mode_DS=NumbersDataset_nb_tasks_id=23f26144-1730-11ec-806c-506b4b3bff98.pth.tar
EVAL on prev heads: /scratch1/amala/DUA/results/train/NumbersDataset_nb_tasks/IMM/MLP_cl_100_100/gridsearch/demo/IMM_lr=0.001_bs=20_epochs=10_lmbL2trans=0.001_alpha=1/task_2/TASK_TRAINING/modelmerge_user1_mode_DS=NumbersDataset_nb_tasks_id=23f26144-1730-11ec-806c-506b4b3bff98.pth.tar
Overall Accuracy: 90.7823613086771
=> Testing model trained up to and including TASK 5
Testing on model = /scratch1/amala/DUA/results/train/NumbersDataset_nb_tasks/IMM/MLP_cl_100_100/gridsearch/demo/IMM_lr=0.001_bs=20_epochs=10_lmbL2trans=0.001_alpha=1/task_5/TASK_TRAINING/modelmerge_user1_mode_DS=NumbersDataset_nb_tasks_id=23f26144-1730-11ec-806c-506b4b3bff98.pth.tar
EVAL on prev heads: /scratch1/amala/DUA/results/train/NumbersDataset_nb_tasks/IMM/MLP_cl_100_100/gridsearch/demo/IMM_lr=0.001_bs=20_epochs=10_lmbL2trans=0.001_alpha=1/task_2/TASK_TRAINING/modelmerge_user1_mode_DS=NumbersDataset_nb_tasks_id=23f26144-1730-11ec-806c-506b4b3bff98.pth.tar
Overall Accuracy: 91.2375533428165
TESTING ON TASK 3
Testing Dataset = /scratch1/amala/DUA/data/datasets/MNIST+SVHN/user_testsubset_SubsetSVHN_targets[4, 5].pth
=> Testing model trained up to and including TASK 3
Testing on model = /scratch1/amala/DUA/results/train/NumbersDataset_nb_tasks/IMM/MLP_cl_100_100/gridsearch/demo/IMM_lr=0.001_bs=20_epochs=10_lmbL2trans=0.001_alpha=1/task_3/TASK_TRAINING/modelmerge_user1_mode_DS=NumbersDataset_nb_tasks_id=23f26144-1730-11ec-806c-506b4b3bff98.pth.tar
EVAL on prev heads: /scratch1/amala/DUA/results/train/NumbersDataset_nb_tasks/IMM/MLP_cl_100_100/gridsearch/demo/IMM_lr=0.001_bs=20_epochs=10_lmbL2trans=0.001_alpha=1/task_3/TASK_TRAINING/modelmerge_user1_mode_DS=NumbersDataset_nb_tasks_id=23f26144-1730-11ec-806c-506b4b3bff98.pth.tar
Overall Accuracy: 65.14472075010191
=> Testing model trained up to and including TASK 4
Testing on model = /scratch1/amala/DUA/results/train/NumbersDataset_nb_tasks/IMM/MLP_cl_100_100/gridsearch/demo/IMM_lr=0.001_bs=20_epochs=10_lmbL2trans=0.001_alpha=1/task_4/TASK_TRAINING/modelmerge_user1_mode_DS=NumbersDataset_nb_tasks_id=23f26144-1730-11ec-806c-506b4b3bff98.pth.tar
EVAL on prev heads: /scratch1/amala/DUA/results/train/NumbersDataset_nb_tasks/IMM/MLP_cl_100_100/gridsearch/demo/IMM_lr=0.001_bs=20_epochs=10_lmbL2trans=0.001_alpha=1/task_3/TASK_TRAINING/modelmerge_user1_mode_DS=NumbersDataset_nb_tasks_id=23f26144-1730-11ec-806c-506b4b3bff98.pth.tar
Overall Accuracy: 65.87851610273135
=> Testing model trained up to and including TASK 5
Testing on model = /scratch1/amala/DUA/results/train/NumbersDataset_nb_tasks/IMM/MLP_cl_100_100/gridsearch/demo/IMM_lr=0.001_bs=20_epochs=10_lmbL2trans=0.001_alpha=1/task_5/TASK_TRAINING/modelmerge_user1_mode_DS=NumbersDataset_nb_tasks_id=23f26144-1730-11ec-806c-506b4b3bff98.pth.tar
EVAL on prev heads: /scratch1/amala/DUA/results/train/NumbersDataset_nb_tasks/IMM/MLP_cl_100_100/gridsearch/demo/IMM_lr=0.001_bs=20_epochs=10_lmbL2trans=0.001_alpha=1/task_3/TASK_TRAINING/modelmerge_user1_mode_DS=NumbersDataset_nb_tasks_id=23f26144-1730-11ec-806c-506b4b3bff98.pth.tar
Overall Accuracy: 65.79698328577253
TESTING ON TASK 4
Testing Dataset = /scratch1/amala/DUA/data/datasets/MNIST+SVHN/user_testsubset_SubsetSVHN_targets[6, 7].pth
=> Testing model trained up to and including TASK 4
Testing on model = /scratch1/amala/DUA/results/train/NumbersDataset_nb_tasks/IMM/MLP_cl_100_100/gridsearch/demo/IMM_lr=0.001_bs=20_epochs=10_lmbL2trans=0.001_alpha=1/task_4/TASK_TRAINING/modelmerge_user1_mode_DS=NumbersDataset_nb_tasks_id=23f26144-1730-11ec-806c-506b4b3bff98.pth.tar
EVAL on prev heads: /scratch1/amala/DUA/results/train/NumbersDataset_nb_tasks/IMM/MLP_cl_100_100/gridsearch/demo/IMM_lr=0.001_bs=20_epochs=10_lmbL2trans=0.001_alpha=1/task_4/TASK_TRAINING/modelmerge_user1_mode_DS=NumbersDataset_nb_tasks_id=23f26144-1730-11ec-806c-506b4b3bff98.pth.tar
Overall Accuracy: 56.95695695695696
=> Testing model trained up to and including TASK 5
Testing on model = /scratch1/amala/DUA/results/train/NumbersDataset_nb_tasks/IMM/MLP_cl_100_100/gridsearch/demo/IMM_lr=0.001_bs=20_epochs=10_lmbL2trans=0.001_alpha=1/task_5/TASK_TRAINING/modelmerge_user1_mode_DS=NumbersDataset_nb_tasks_id=23f26144-1730-11ec-806c-506b4b3bff98.pth.tar
EVAL on prev heads: /scratch1/amala/DUA/results/train/NumbersDataset_nb_tasks/IMM/MLP_cl_100_100/gridsearch/demo/IMM_lr=0.001_bs=20_epochs=10_lmbL2trans=0.001_alpha=1/task_4/TASK_TRAINING/modelmerge_user1_mode_DS=NumbersDataset_nb_tasks_id=23f26144-1730-11ec-806c-506b4b3bff98.pth.tar
Overall Accuracy: 58.85885885885886
TESTING ON TASK 5
Testing Dataset = /scratch1/amala/DUA/data/datasets/MNIST+SVHN/user_testsubset_SubsetSVHN_targets[8, 9].pth
=> Testing model trained up to and including TASK 5
Testing on model = /scratch1/amala/DUA/results/train/NumbersDataset_nb_tasks/IMM/MLP_cl_100_100/gridsearch/demo/IMM_lr=0.001_bs=20_epochs=10_lmbL2trans=0.001_alpha=1/task_5/TASK_TRAINING/modelmerge_user1_mode_DS=NumbersDataset_nb_tasks_id=23f26144-1730-11ec-806c-506b4b3bff98.pth.tar
EVAL on prev heads: /scratch1/amala/DUA/results/train/NumbersDataset_nb_tasks/IMM/MLP_cl_100_100/gridsearch/demo/IMM_lr=0.001_bs=20_epochs=10_lmbL2trans=0.001_alpha=1/task_5/TASK_TRAINING/modelmerge_user1_mode_DS=NumbersDataset_nb_tasks_id=23f26144-1730-11ec-806c-506b4b3bff98.pth.tar
Overall Accuracy: 67.91641057160417
FINISHED testing for user: 1
USER1 results: {'acc': {0: [94.9137679041216, 89.09675533469746, 87.0798012277112, 85.8228588132125, 85.09207833966677], 1: [89.55903271692745, 90.49786628733997, 90.7823613086771, 91.2375533428165], 2: [65.14472075010191, 65.87851610273135, 65.79698328577253], 3: [56.95695695695696, 58.85885885885886], 4: [67.91641057160417]}, 'forgetting': {0: [5.817012569424136, 7.833966676410398, 9.090909090909093, 9.821689564454829], 1: [-0.9388335704125268, -1.2233285917496488, -1.6785206258890497], 2: [-0.7337953526294427, -0.6522625356706158], 3: [-1.9019019019019012], 4: []}}
**************************************** ITERATION 2 ****************************************
[WARN] SKIPPING iteration, as no randomness introduced.
**************************************** ITERATION 3 ****************************************
[WARN] SKIPPING iteration, as no randomness introduced.
**************************************** ITERATION 4 ****************************************
[WARN] SKIPPING iteration, as no randomness introduced.
**************************************** ITERATION 5 ****************************************
[WARN] SKIPPING iteration, as no randomness introduced.
Saved exp results to: /scratch1/amala/DUA/results/test/results/NumbersDataset_nb_tasks/IMM_mode/MLP_cl_100_100/demo/subset=[0, 1, 2, 3, 4]/IMM_lr=0.001_bs=20_epochs=10_lmbL2trans=0.001_alpha=1/evalresults_IMM_mode_user1.pth