Reduce validation loss in training about platalea HOT 5 CLOSED

First try: dropout = 0.1. Run: lunar-serenity-26.

from platalea.

Promising, surely better results than with 0 dropout. Now trying dropout = 0.2: dark-serenity-27. Note: this is only dropout in the transformer encoder layer.

from platalea.

Also tried 0.5 (worthy-sunset-28). Both cases: difference between validation loss and step loss stays slightly lower initially than in the lower dropout factor runs.

However, also running logical-butterfly-29 and there we see that after some more training the validation loss still starts climbing again, whereas the training loss more or less completely vanishes, so again overfit on training set at some point. The performance is better though, and still climbing at this moment (epoch 130 of 300), so let's see where it goes.

from platalea.

We did a sweep again over dropout values and again values between 0.2-0.5 seem to perform best. However, validation loss still increases in all runs after a certain time.

As already mentioned in this report, we should probably look into more regularization options to correct for this overfitting on training data. See #61.

from platalea.

I think we solved the validation loss (overfitting) issue during our regularization sweeps (https://wandb.ai/spokenlanguage/platalea_transformer/reports/Jan-29-Project-Update-Regularization-rates-conclusion--Vmlldzo0MzY3MDg).

from platalea.