Support for Albert and Swin/ViT about deepspeed-mii HOT 1 OPEN

@larry-fuy there are no immediate plans to support these models, but we are very welcome to outside contributions. Our documentation could certainly be improved in terms of describing how to add new model support, so let me provide a starting point here and I would be happy to help in the process.

In general, to support a model we need to have support for kernel injection on the DeepSpeed-inference side (but we can also run models without kernel injection if we only want to use ZeRO). For example, we support automatic kernel injection with Bert, GPT-J, etc. models with DeepSpeed-inference. Kernel injection can also be achieved with custom injection policies. Check out this example from our unit tests on how to do this:
https://github.com/microsoft/DeepSpeed/blob/62c071e016aca6dbf12695bdb680721dcf3500ad/tests/unit/inference/test_inference.py#L408

Once we have this support on the DeepSpeed-inference side, adding the model to MII should be as easy as expanding the list of supported model types and tasks.

Please let me know if I can clarify anything further.

from deepspeed-mii.