Dear Dr. Klein,

Many thanks for the benchmark datasets on CNN and Fully-connected networks. I'm trying to reproduce some of the results of the fully connected networks on protein structure uci dataset using your script "train_fcnet.py". However, I got quite different performance (valid_mse=0.5) as compared to the results reported in the tabular benchmark dataset (valid_mse=0.3) on the same configuration/hyperparamters.

Is it possible for you to share the train/valid/test.npy files you used for the fcnet tabular benchmark dataset? Because that's the only variant in my script (btw, I also remove the redundant feature and normalise the data as instructed in your paper)?

I'd be really grateful to hear your reply. Once again, thanks for the datasets.

Best,
Robin
Oxford

I suspect that in this line you should have budget and not the constant 100 in the divisor:

hi,
in data_collection script (https://github.com/automl/nas_benchmarks/blob/master/data_collection/train_fcnet.py), data file "%s_train_data.npy" is specific, but how to obtain these data file?

Hi there,
thanks for both releasing NAS-Bench-101 and the code you used to benchmark on it, I believe it's a really good step towards fairer comparison of NAS methods.

It would be really useful if you also released the scripts (including the hyper-parameters) used to generate the main plot (for example Fig.7).

Would that be possible? Thanks

Hi, Thank you for this work! I would like to ask if I want to reproduce the paper in the paper, do I run the code in experiments_script directly, and the json file obtained is the result described in the paper? Are there any details that I need to pay attention to or need to modify?Looking forward to your reply.

Hi,

This is wrt fcnet_tabular_benchmarks.

The Regularized Evolution script evaluates accuracy by subtracting the objective_function() evaluation from 1.

However, for certain benchmarks the MSE returned is greater than 1, which can make the accuracy negative. Is this evaluation correct?
Accuracies are computed here and here.

One way to verify this may be so:

b = FCNetSliceLocalizationBenchmark(data_dir)
cs = b.get_configuration_space()

for i in range(1000):
     config = cs.sample_configuration()
     value, _ = b.objective_function(config)
     if value > 1:
         print(config)
         break
for i in range(6):
    value, _ = b.objective_function(config)
    print(1 - value)

Hello there,

Thank you for making this benchmark public. I have a question regarding the dataset of cifarA, cifarB, cifarC, and they all inherit a class that wraps nasbench. So my question is, what's the difference of cifarA, B, C, and their purpose? Thank you.