can't find saved weights after training about transformersum HOT 7 CLOSED

Have you checked the directory - trained_models/lightning_logs/version_[VERSION_NUM]/checkpoints/epoch=[X].ckpt ?

from transformersum.

there's nothing in trained_models.

from transformersum.

I think this had worked in the past with a different model, but it is not working anymore

from transformersum.

Yeah I can't see the directory now either but I see this trained_models.ckpt file in the directory I ran it from.

It contains the model saved state and other config stuff, although I used this flag --use_custom_checkpoint_callback during training so that might be the case as well

In [3]: t = torch.load("trained_models.ckpt")

In [4]: t.keys()
Out[4]: dict_keys(['epoch', 'global_step', 'pytorch-lightning_version', 'callbacks', 'optimizer_states', 'lr_schedulers', 'state_dict', 'hparams_name', 'hyper_parameters'])

In [5]: t['epoch']
Out[5]: 1

from transformersum.

ok let me try again with that flag

from transformersum.

I have made this work by setting use_logger tensorboard -- otherwise the checkpoint is saved in the wandb folder -- even though there doesn't appear to be tensorboard logging support. Also, I needed to set --use_custom_checkpoint_callback to get a checkpoint at the end of every epoch -- otherwise it only saves 1 checkpoint, and based on the label it is epoch0 -- if there's only 1 checkpoint I'd rather it be the last epoch.

from transformersum.

@moyid Try setting --no_wandb_logger_log_model and not setting the --weights_save_path.

from transformersum.