Automatically save best model during training about aigt HOT 1 OPEN

Finding the best model is not as simple as saving the one that has minimum loss function. Sometimes when training for extremely long (thousands of epochs), the model can learn a better (more general) representation of the data without decreasing the loss function. I couldn't find where I learned this, but it was related to this theory: https://medium.com/@MITIBMLab/estimating-information-flow-in-deep-neural-networks-b2a77bdda7a7

Saving the model regularly is generally a good practice. We could add an option like "model_save_frequency". E.g. if it's 5 then the model would be saved after every 5 epochs using names like model_005, model_010, etc. And we could save on the wandb report all the metrics for each saved model.

I also had positive experience in the past training for a few hundred more epochs after it seemed like the metrics did not improve.

from aigt.