Thanks for the great work again!

I've successfully run WS-GM/exp_scripts/rsws-ws-201.sh, and now I obtained four checkpoints of supernets, e.g., supernets_0/1/2/3.pt.

I want to reproduce the results of Table 7, but here are some questions:

1. what do you mean by top 0.2%/0.5%/1% architectures in NAS-Bench-201? Are these architectures ranked according to their real test accuracy or proxy accuracy?
2. Table 7 shows that you obtained16 sub-Supernets after partition. However, I only got 4 sub-Supernets.
3. Could you please provide the source code and model weights for reproducing the spearman score on the top1% models?

Hello,

I'm trying to run the provided code but the results are inconsistent with the paper.

NASBench-201 space on CIFAR-10, after running train_search.py,
With DARTS: I get train: 86.59 and test: 90.44
With SNAS: I get valid: 89.38 and test: 92.76.

Both experiments are conducted on seed = 0.

Could you please provide some information about the version (python, pytorch,..) or some other possible solutions?
Thank you.

Thanks for the great work.

I wonder how you train the sub-supernets before splitting the supernet by grads?

Let's take NASBench201 as an example, say we have a sub-supernet with encodings of

 tensor([[1., 0., 1., 1., 0.],
        [1., 1., 1., 1., 1.],
        [1., 1., 1., 1., 1.],
        [1., 0., 1., 1., 0.],
        [1., 1., 1., 1., 1.],
        [1., 1., 1., 1., 1.]], device='cuda:0')

Are all operations with the value of 1 involved in the forward and backward processes? Or do you randomly sample only one operation for each edge for each training batch?

In WS-GM/nasbench201/train_search.py the train function has
input, target = next(iter(train_queue))
inside the batches for loop. This creates an iter object every batch which significantly slows down training. It also makes it so 1 epoch doesn't loop over the entire dataset. Because iter(train_queue) is initialized every batch, it grabs the first sample, then reinitializes, grabs the first sample etc. You get a completely random sample from your dataset, but there is no guarantee that it is disjoint.