Hi

Thanks for the paper and code.

I noticed that a few argument values in search/src/args.py differ from what is mentioned in the paper like max_weight_decay, mul_factor.

Also if I understood it right, in the paper most plots seem to indicate that overfitting is reduced with drop_path_prob >= 0.5, yet the default value of it in code is given 0.2

The value of "factor" parameter (the threshold for eigenvalues) is also different in the paper than code.

Could you please clarify about the good/default value of all these 3 parameters as per your experiments?

Thank You!

Hi, I appreciate for sharing your work including the paper and this open-source code.

I have a question about measuring "performance drop" in 4.2 section of the paper (figure 5(b)).

Let me explain my concern with an example. My understanding of RobustDARTS is as follows:

1. validation accuracy of models are measured after search phase (i.e., after 50 epochs of search)
2. weighting in mixed operation (i.e., softmax output of alpha) is not one-hot before discretizing alpha, and is converted into one-hot vector after discretizing to leave one operation per edge.
   ex) [one edge] alpha after 50 epoch search = [0.1 0.4 0.2 0.1]
   --> softmax(alpha) = [0.2245 0.3030 0.2481 0.2245]
   --> this is used for weighting of mixed op before discretizing.
   • after discretizing --> the weighting of mixed op is converted into an one-hot vector: [0 1 0 0].

Please check if my understanding is right.

If it is, the code of "discrete == True" in forward() of Network class in model_search.py seems to have an error.
In your code, weighting is [0.1 0.4 0.2 0.1] if discrete (without softmax activation), otherwise [0.2245 0.3030 0.2481 0.2245] (with softmax activation).
In my opinion, if discrete, the weighting should be [0 1 0 0].

Please check this issue.
If I am misunderstanding the discretizing process, I would like to ask for a clearer explanation of the discretizing step and the measurement of "performance drop."

Thank you.

Hi,

Noticed you are monitoring 1st-order derivatives of loss versus architecture parameters as well when compute_hessian is activated. Is it to make sure the 1st-order derivatives are actually close to zero so the eigenvalues of hessian would be the global curvatures?

If so, what's the mechanism do you apply to judge if 1st-order derivatives are close to zero? Thanks very much!

Best,
Bolian

Hi, I found something weird in src/search/train_search.

In line 161, if args.compute_hessian, ev is assigned -1.
But i think it should be

if not args.compute_hessian:
    ev = -1
else:
    ev = la_tracker.ev[-1]

because la_tracker.ev is only appended something in line 421 when args.compute_hessian is True.
If I directly run your code, a list index out of range error occurs.

I am not sure if I understand it right.
Thank you!

How to calculate the number of layers of the network （robustDARTS）?

Hi

In train_search.py, there seems an intention to be strengthening the data augmentation along with the architecture search progress by increasing the probability to activate Cutout.

  if args.drop_path_prob != 0:
    model.drop_path_prob = args.drop_path_prob * epoch / (args.epochs - 1)
    train_transform.transforms[-1].cutout_prob = args.cutout_prob * epoch / (args.epochs - 1)

However, the variable in Cutout is prob rather than cutout_prob; therefore, it looks like Cutout is always activated if args.cutout is set as True. Should the code be revised? Thanks.

  class Cutout(object):
    def __init__(self, length, prob=1.0):
    self.length = length
    self.prob = prob