Embed LR schedule and initialization with the model about vision HOT 11 OPEN

Weights can be initialized in model's __init__, so it has nothing to do with the imagenet example, right?

As for lr schedule, I think we can just do sth like

if hasattr(model, 'lr_schedule'):
    lr = model.lr_schedule(epoch)
else:
    lr = args.lr * (0.1 ** (epoch // 30))

from vision.

I've been trying to put weight initialization as part of the model, since it often seem particular to the type of architecture. I added it to the ResNet definition and I'm going to add it to the VGG model def:
https://github.com/pytorch/vision/blob/master/torchvision/models/resnet.py#L112

I'm not sure about learning rate schedule. It seems awkward to put it as part of the model definition, but as you point out, hard coding it in the ImageNet example isn't ideal either

from vision.

@apaszke @colesbury Thanks, I will do the same for SqueezeNet. Are there pre-defined initialization functions/classes for popular initialization schemes (e.g. like in Neon or Keras)?

@colesbury I think ideally we should provide a default learning schedule as a part of torch.vision models and let users override it via command-line arguments.

from vision.

Yes, we should add them in nn somewhere.

from vision.

What do you think about this kind of regime inside of the model?
https://github.com/eladhoffer/convNet.pytorch/blob/master/models/alexnet.py

from vision.

Can't open it, are you sure the link is correct and the repo is public?

from vision.

How about https://raw.githubusercontent.com/eladhoffer/convNet.pytorch/master/models/alexnet.py
The repo is public

from vision.

That's one way to approach it, but I'm not sure if it's the most convenient one. Having a function that returns an optimizer for a given epoch seems more powerful.

from vision.

Is there/will there be a nice way to adapt the learning rate or momentum but keep other state in the optimizer, i.e. for Adam

from vision.

Should this issue be moved to the pytorch repo instead?

from vision.

As you, guys, speak also here about weight initialization, what about DenseNet if we want to use a not pretrained model ?
According to Caffe official implementation, convolutions are initialized with something like kaiming_normal.

from vision.