Estimated standard deviations calculation about optimizer HOT 5 CLOSED

Hey Tal, I dug into the logs of my previous experiments, and I found that I was using 1 model for each subrun. To replicate my calculation of standard deviation, target_model_count_subrun should be set to 1.

Those arguments were added while I was cleaning the code after the experiment have finished, and I apologize for the inconsistency.

To calculate the standard deviation of the estimated mean, I used the following formula

$s=\sqrt{\frac{\sum_{i=1..n}(x_i - \bar{x})^2}{n}}, s_{mean} = \frac{s}{\sqrt{n}}$ where $n$ = 175.

from optimizer.

Hi,
Thanks for your quick response.
So if n=175 that means you have chosen 'target_model_count'=175 and not 'target_model_count'=200 as in current 'mnist_guess.yaml'.
To sum up, i understand that your reported experiment at table 2 in the paper was made with: target_model_count=175, target_model_count_subrun=1. Is that correct?

from optimizer.

Yes, that is correct.

from optimizer.

I am closing this issue since it has been resolved.

from optimizer.

Hi,
I am sorry for re-opening this issue but something in your calculation of standard deviation of the estimated mean looks odd to me.
From my understanding, for each combination of (num_train_samples, loss_bin) you calculate 's' using 175 test accuracies that you found during the run.
Why do you calculate 's_mean' and treats it like the standard deviation of the estimated mean?
Is 's' not the result we are looking for?

Thanks!

from optimizer.