Fix model surgery so `Event.INIT` can be removed from Trainer `__init__` about composer HOT 4 CLOSED

@anisehsani curious about the status of this one? If we can get this merged in, then I think we should be able to remove Event.INIT and move the Event.TRAINING_START right before all the ddp wrapping begins. This would help clean up the trainer, as then no events would fire in __init__.

from composer.

Not positive, but I think the path to getting this working cleanly is to make it easy to add_param_group to the optimizer after its construction (but still before ddp). And to default to adding all the model's params to a group. As long as the optimizer's defaultdict has the same hparams as would have been used when passing params in at __init__, this should end up with equivalent behavior (in theory).

The more comprehensive solution, which would also enable dynamic factorization, quantization, channel pruning (?), etc, is to work around DDP needing the params and their shapes frozen, which would solve this as a special case. But that's probably a lot harder.

from composer.

This issue presents an interesting API challenge:

optimizers depends on the model
schedulers depends on the optimizers and steps_per_epoch

It may make sense to wait until after #146 is implemented, as then steps_per_epoch would no longer required to initialize schedulers

from composer.

Closing for #249 and #157

from composer.