Unreproducible Results given seed about syne-tune HOT 9 CLOSED

@ArlindKadra You refer to examples/launch_simulated_benchmark.py in a pretty old branch benchmark_tutorial. Is this intended? I will look whether the problem happens in current main.

from syne-tune.

Look at examples/launch_simulated_benchmark.py, it is no surprise this is not reproducible. You can see that objectives_evaluations are generated at random, so the benchmark itself will be different every time it is run.
We can fix the seed for this example.

from syne-tune.

Maybe it would be better to look at your example.

from syne-tune.

Ups, that is a very correct observation that I forgot when I quickly tried to replicate my problem on the example. Hmm, maybe dump the objective evaluation data and load the same backup with the different seed for running?

from syne-tune.

If it works well, then I guess the problem could be from my end, maybe the script that I quickly hacked together.

from syne-tune.

@ArlindKadra You refer to examples/launch_simulated_benchmark.py in a pretty old branch benchmark_tutorial. Is this intended? I will look whether the problem happens in current main.

Not intended.

from syne-tune.

One thing to check: If you use random search and fix the seed, then you should see exactly the same configs being proposed (set debug_log=True for full output). But of course, you also need to make sure that the metric values from the blackbox are the same.

from syne-tune.

I am pretty confident results should be reproducible (unless GP-BO is involved), because I've been looking at this several times.

from syne-tune.

Thanks for taking the time to take a look at it. You were right, I was extrapolating some curves and because of some bad logic there was noise involved in the last epochs which in turn changes the incumbent trajectory.

Framework works perfect, results are reproducible :).

from syne-tune.