[Warning] the current text generation call will exceed the model's predefined maximum length (4096). about spacy-llm HOT 2 OPEN

Hi @yileitu, all model parameter are forwarded to transformers that handles the model. In most cases there is a max_length or max_new_tokens parameter you can set:

[components.llm.model]
@llm_models = "spacy.Llama2.v1"
name = "Llama-2-7b-hf"
max_length = 8192  # or any other value you want to set

from spacy-llm.

Hi @rmitsch. Thanks for your replies. However this does not work.

Config validation error
llm.model -> max_new_tokens	extra fields not permitted
{'@llm_models': 'spacy.Llama2.v1', 'name': 'Llama-2-7b-hf', 'max_new_tokens': 10000}

Neither does max_length.

from spacy-llm.