Failed trials have out of date metrics about syne-tune HOT 5 CLOSED

When a trial is stopped by a scheduler such as ASHA, a stopping signal is sent to the trial.
If some metrics are emitted before this signal is seen, those metrics are not communicated to the scheduler and not shown in summary table (however, they are taken into account when showing the best trial at the end).

The reason for not showing this result to schedulers is that most of them would not support acting on results seen after they emit a stopping decision.

from syne-tune.

One thing we could do is to report results in this table even after stopping decisions even though scheduler would not see those results.

from syne-tune.

Thanks for your explanation! I think in some cases it could help to report. For example in my case just some profiling printouts failed at the end of training for a few trials, but only after diving deeper into it did I realize that those ones actually performed the best.

from syne-tune.

Hello @austinmw , can we close this issue? I think that whenever a trial script fails, Syne Tune will not consider future results coming from there, this is probably a safe behaviour, right?

from syne-tune.

Sure, that makes sense to me, thanks.

from syne-tune.