When running mii.serv, it keeps print waiting for server to start. about deepspeed-mii HOT 6 CLOSED

This is the model dir and model file

,I use vLLm to run a server is OK, code like this.

nohup python -m vllm.entrypoints.openai.api_server --model="/data/python_workspace/text-generation-webui/models/neural-chat-7b-v3-1" --trust-remote-code --port=5000 --host="0.0.0.0" > output.log 2>&1 &

from deepspeed-mii.

@cninnovationai could you please try loading the model with pipeline and let me know if that works? Thanks

import mii
pipe = mii.pipeline("/data/python_workspace/text-generation-webui/models/neural-chat-7b-v3-1/")
response = pipe("test")

from deepspeed-mii.

@cninnovationai could you please try loading the model with pipeline and let me know if that works? Thanks

import mii
pipe = mii.pipeline("/data/python_workspace/text-generation-webui/models/neural-chat-7b-v3-1/")
response = pipe("test")

[2023-12-26 14:40:25,529] [INFO] [real_accelerator.py:161:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2023-12-26 14:40:32,301] [INFO] [comm.py:637:init_distributed] cdb=None
[2023-12-26 14:40:32,305] [INFO] [comm.py:668:init_distributed] Initializing TorchBackend in DeepSpeed with backend nccl
[2023-12-26 14:40:32,360] [INFO] [engine_v2.py:82:init] Building model...
Using /home/bruce/.cache/torch_extensions/py310_cu121 as PyTorch extensions root...

always stay here,No further progress, gpu memory usage is 0

Do you know how to solve this problem? @mrwyattii

from deepspeed-mii.

@cninnovationai it looks like there is a lock file in your torch cache that is causing the issue. Try deleting your torch cache and run again:
rm -rf /home/bruce/.cache/torch*

from deepspeed-mii.

Cleared torch cache and worked for me

from deepspeed-mii.

after rm -rf /home/bruce/.cache/torch*,it work

from deepspeed-mii.