name 'vllm' is not defined about llm-blender HOT 3 CLOSED

Hi @siddiquelatif , I have tested this bug, and removed this comment in the latest version,
before

def best_of_n_generate(
        self,
        model: Union[transformers.PreTrainedModel, vllm.LLM], # Union[transformers.PreTrainedModel, vllm.LLM]
        model_tokenizer:transformers.PreTrainedTokenizer,
...

after

def best_of_n_generate(
        self,
        model, # Union[transformers.PreTrainedModel, vllm.LLM]
        model_tokenizer:transformers.PreTrainedTokenizer,
...

so there will be bugs if vllm is not installed. However, VLLM models are also supported if you want. Please try to reinstall the LLM-blender package. This bug will be fixed.

from llm-blender.

Can you try deleting the variable type comments Union[transformers.PreTrainedModel, vllm.LLM]? Will there still be errors?

from llm-blender.

Thanks @dongfu Jiang,

Yes, its working. Thanks again. :)

from llm-blender.