About fixed weights from a grid search experiments setting about multi-task-learning-pytorch HOT 3 CLOSED

Hi. Thanks for the interest.
If you want to find a quick way of setting the weights. I would suggest to track the task-specific losses over a few iterations/epochs for every task. Then adjust the weights on the tasks a bit, so all of the losses are of similar magnitude.
For example, if we solve two tasks, with weights 1.0, and the average loss of task 1 is 100 times the average loss of task 2.
Then I would train with a weight * 100 for the second task.
Also, if you use a multi-task baseline, I find that Adam can sometimes achieve a bit higher numbers compared to SGD.

Ofcourse, you might get even better results if you further try to adjust the weights through trial-and-error. However, I find that the proposed method works well in practice.

Good luck.

from multi-task-learning-pytorch.

Hi @SimonVandenhende,

Thank you so much for the suggestions. For the Adam optimizer, does amsgrad parameter need to be set to True?

Best.

from multi-task-learning-pytorch.

Hi @ganlumomo

I only explored with the regular Adam optimizer and SGD (see the get_optimizer function in utils/common_config.py).
It should be possible to get good results when combining these optimizers with properly initialized weights (following the procedure outlined above).

from multi-task-learning-pytorch.