Hi,

I am trying to reproduce the results of RobNets and got a little confused with your SVHN implementation.

It seems like the RobNet used for SVHN has 100 classes.

Also the provided checkpoint has 100 classes I think. I was trying to find details with the datasets in your paper but failed to find anything that describes this discrepancy.

Thanks.

Hi, @gmh14 , thanks for our implementation, I notice that you re-implement the SVHN dataset in the dataset.py, kindly want to know what is the point of rewriting the dataset rather than using the torchvision datasets?

Thanks,

Hi! I appreciate your good work!

Since the supernet is not release, is there anyway we can replicate your results on correlations between architecture density and their robustness? It would be very useful if you could release some of the specific architectures that you sampled. Even a list of 100 out of the total 1000 genotypes you sampled would be very helpful.

Thanks!

I am new to the field of adversarial attacks. I noticed that the natural accuracies of ResNet18 and ResNet50 in CIFAR10 are 78.38% and 79.15% respectively, which are much lower than other pre-trained models. So, what is the reason for this? Is it because of adversarial training using PGD? I also found a lack of preprocessing operations, e.g., normalization operations, prior to training the models.

Excuse me, we fail to adversarially train RobNet family on tiny-imagenet from scratch that the accuracy and adversarial accuracy on validation set don't rise at all. Only the learning rate decay strategy is mentioned in your article. Could you please tell me the whole detailed training configuration on tiny-imagenet?

We can use this project with pytorchcount to calculate these parameters? thank you very much!

Hello, I want to collect some networks with defensive properties and different structures. Can this code achieve this?
I'm still working on the train part of the Robnets code.But I was wondering if you could give me an answer to the former question that Robnets didn't release the training code for the super net.
Does this affect the extraction of subnetworks?

I have been trying to do adversarial training on MobileNetV2 for CIFAR10 using a standard codebase, and my natural/robust accuracies are no where near the reported numbers in Table 7 of the supplementary material.
Could you please share the exact training hyperparameters for the MobileNetV2 model? or the model parameters if that is possible. Thanks.

I am new to the field of adversarial attacks. I noticed that the natural accuracies of ResNet18 and ResNet50 in CIFAR10 are 78.38% and 79.15% respectively, which are much lower than other pre-trained models. So, what is the reason for this? Is it because of adversarial training using PGD? I also found a lack of preprocessing operations, e.g., normalization operations, prior to training the models.

Hi, I am trying to test the result of your checkpoint "RobNet_free_SVHN.pth.tar". Someone has pointed out your SVHN implementation mistake. Therefore, to use the checkpoint, we directly take the first 10 dimensions of the weights and bias of the fully connected layer. And in order to avoid the difference caused by the attacker's implementation, we directly use your PGD attacker.

Results

However, result we got is quite different than that reported in your article.

Ours: acc clean = 94.09%, acc adv = 60.45%.
Your report: acc adv = 55.59%.

Could you please tell me why?
Is it caused by the mistake mentioned above?

Our code

import random

import numpy as np
import torch
import torch.utils.data
from torch import nn
import torch.nn.functional as F
from torchvision import transforms, datasets

import models
import architecture_code
import utils

class AttackPGD(nn.Module):
    def __init__(self, model, config):
        super(AttackPGD, self).__init__()
        self.model = model
        self.rand = config['random_start']
        self.step_size = config['step_size']
        self.epsilon = config['epsilon']
        self.num_steps = config['num_steps']
        self.attack = config['attack']

    def forward(self, inputs, targets):
        if not self.attack:
            return self.model(inputs), inputs
        x = inputs.detach()
        if self.rand:
            x = x + torch.zeros_like(x).uniform_(-self.epsilon, self.epsilon)
        for i in range(self.num_steps):
            x.requires_grad_()
            with torch.enable_grad():
                logits = self.model(x)
                loss = F.cross_entropy(logits, targets, size_average=False)
            grad = torch.autograd.grad(loss, [x])[0]
            x = x.detach() + self.step_size * torch.sign(grad.detach())
            x = torch.min(torch.max(x, inputs - self.epsilon), inputs + self.epsilon)
            x = torch.clamp(x, 0, 1)
        
        return self.model(x), x

def svhn():
    test_transform = transforms.Compose([transforms.ToTensor()])
    testset = datasets.SVHN(root="/home/zhaojb17/awnas/data/SVHN/", split='test', download=True, transform=test_transform)
    queue = torch.utils.data.DataLoader(testset, batch_size=24, pin_memory=True, num_workers=2)
    return queue

def set_seed(seed):
    # Set seed for system, numpy and pytorch.
    if seed is not None:
        np.random.seed(seed)
        random.seed(seed)
        torch.manual_seed(seed)

set_seed(123)

# use RobNet architecture
net = models.robnet(architecture_code.robnet_free)
net = net.cuda()

# load pre-trained model
utils.load_state('./checkpoint/new_svhn.pt', net)

testloader = svhn()

net.eval()
correct, adv_correct, total, n_batch = 0, 0, 0, len(testloader)

# Set attacker
cfg = { "random_start": True,
        "step_size": 2.0/255,
        "epsilon": 8.0/255,
        "num_steps": 100,
        "attack": True
        }
attacker = AttackPGD(net, cfg)

for i, data in enumerate(testloader):

    images, labels = data
    images = images.cuda()
    labels = labels.cuda()

    outputs = net(images)
    _, predicted = torch.max(outputs.data, 1)
    correct += (predicted == labels).sum().item()
    
    outputs = attacker(images, labels)[0]
    _, predicted = torch.max(outputs.data, 1)
    adv_correct += (predicted == labels).sum().item()
    
    total += len(images)
    
    print("{}/{}: {} {}".format(i, n_batch, correct / total, adv_correct / total))

correct = correct / total
adv_correct /= total

if it possible to share the RobFree's .pt file with me, i want to do some evalution on it. thanks a lot.

How do I do archchtecture search on supernet?

Could you please open source the whole training process? I think that would more valuable than this fixed model. Thank you for your time and consideration.

Hi, in your paper, you say "After training the supernet, we randomly sample 1,000 architectures from the supernet and finetune each of them for 3 epochs", how to do this? Constructing a network according to a randomly generated genotype_list and evaluate this network?

Hi,
Congratulations on the work. It seems really interesting.

I was hoping to use your SuperNet approach for comparing the robustness of different architectures (paper mentioned that you compared over 1000 network architectures). I see that there is a Train section in the Readme but I am guessing it is not for the SuperNet.

Is it possible for you to let us know if we can access the trained SuperNet from somewhere or else how can I achieve the task mentioned above. Any help would be appreciated

Excuse me, I want to know the difference between the training time and the memory consumption of the search network process with and without adversarial training? I want to know how much additional time and memory is needed to obtain robnet. I will be grateful for your answers to my questions。

Hi guys,
I want to use a trained super-net for my experiments. Your RobNet_large_v1_cifar10.pth seems to be an option. However, I am afraid of that you have fine-tuned the super-net w.r.t. the architecture hard-coded in architecture_code.py. I read the README but is still not sure.
Thanks!

Hi,

I am trying to find the checkpoint file to reload weights of the RobNet_Free model using the following command:
utils.load_state('./checkpoint/RobNet_free_cifar10.pth.tar', net)

But, I can't locate the checkpoint file anywere on github..
If you have shared the checkpoint file already, would you mind pinpointing me to its location?

Otherwise, do I need to train RobNet_ free from scratch using your training code and reuse weights once the model converges, or
do you plan on releasing the checkpoint files anytime soon?