[Bug] DeepSpeed Inference Does not Work with LLaMA (Latest verison) about deepspeedexamples HOT 3 OPEN

Would be nice if someone can tell which version will work

from deepspeedexamples.

Hi @allanj I don't think we have kernel injection support for llama-2 models. If you remove the --use_kernel flag does the script work?

Additionally, what kind of GPUs are you using? You may be able to utilize DeepSpeed-MII to run the llama-2 model and get significant improvements to inference performance if you have GPUs with compute capability >=8.0:

import mii

client = mii.serve("meta-llama/Llama-2-7b-hf", tensor_parallel=8)
response = client("test prompt")

from deepspeedexamples.

Yes. Removing the --use_kernel make it work.

Yeah, I realize the DeepSpeed FastGen. Wondering, how does it support the batch size? Or I simply make a for loop about that

from deepspeedexamples.