data-drone / anz_llm_bootcamp Goto Github PK

Can you please add the capacity to pass in the current user's Hugging Face token to the use_auth_token parameter during the initialization of tokenizer, model_config, and model in the 0.1_Huggins_Face_basics, Utils, etc. notebooks? It is required to download the model from the Meta website via Hugging Face.

When I load a Llama-2 model and try to get a value back from the query test:

# Test Query 1
query = "What is this document about?"
result = qa.run(query)
print(result)

The response reads:

Input length of input_ids is 2764, but `max_length` is set to 20. This can lead to unexpected behavior. You should consider increasing `max_new_tokens`.
 This

And that's it ~ apart from the warning about max_length being set to 20, I get the single word "This".
I have examined the pipe.model.generation_config and it shows max_length as 4096 (below), and I cannot figure out what the message is referring to, but it doesn't seem to be this max_length !

GenerationConfig {
  "_from_model_config": true,
  "bos_token_id": 1,
  "eos_token_id": 2,
  "max_length": 4096,
  "pad_token_id": 0,
  "temperature": 0.9,
  "top_p": 0.6,
  "transformers_version": "4.29.2"
}