bug: Output text from CompletionChunk is different with tokenizer.decode about openllm HOT 7 OPEN

I will take a look into detokenization incrementally for PyTorch backend.

from openllm.

Hi there, thanks for creating the issue.

Do you have vllm available locally?

from openllm.

Hi

I'm still not able to run this model with vllm backend due to insufficient gpu mem (T4 16g seems not enough)

After some research, I think the root cause of this might be a single complete chinese character may be decoded from multiple token outputs. So decoding to text on every generate iteration is not feasible for Chinese.

from openllm.

Sounds like a orthogonal issue from OpenLLM?

from openllm.

For pytorch backend, it is related to OpenLLM in the implementation of PyTorchRunnable. It might need some way to detect incomplete character on each generation, probably something like what text-generation-inference server had here
OR transformers TextStreamer done here

If vllm backend has this handled, then OpenLLM will be doing fine. But I'm not able to verify it at the moment.

from openllm.

Tried to fix the problem with the text-generation-inference server approach (Related issue: huggingface/text-generation-inference#333)
Please have a look, thanks!

from openllm.

For pytorch backend, it is related to OpenLLM in the implementation of PyTorchRunnable. It might need some way to detect incomplete character on each generation, probably something like what text-generation-inference server had here OR transformers TextStreamer done here

If vllm backend has this handled, then OpenLLM will be doing fine. But I'm not able to verify it at the moment.

FYI, found that vllm had also fix this issue with text-generation-inference approach in this pr vllm-project/vllm#984

from openllm.