Speeding up loading in inference checkpoints about deepspeed-mii HOT 2 OPEN

Hi, you can refer these docs and code examples:

• https://github.com/microsoft/DeepSpeed/tree/master/blogs/deepspeed-fastgen
• https://github.com/microsoft/DeepSpeedExamples/tree/master/inference/mii

for adding new unsupported models:

• https://github.com/microsoft/DeepSpeed/blob/master/deepspeed/inference/v2/model_implementations/AddingAModel.md

for loading local huggingface checkpoints, you can specify the absolute directory path in pipeline

from deepspeed-mii.

Thanks for sending this through - apologies I didn't explain this very well.

What I actually was asking about is if there is a a way to reduce loading time for your own finetuned models from HuggingFace checkpoints, as I'm finding that loading in the default models seems to be much faster.

In particular, this PR microsoft/DeepSpeed#4664 references adding the 'capability to snapshot an engine and resume from it' - hence I was wondering how I may save and load that engine so as to reduce the time taken to load a non-persistent pipeline the first time?

from deepspeed-mii.