LLaMA fails when the input token length is over 1790 tokens about transformers-neuronx HOT 6 CLOSED

By default the tutorial uses top_k=50 sampling, which performs multinomial sampling. If you would like to do deterministic sampling, you can set top_k=1 in your sampling call. This will perform deterministic greedy sampling

from transformers-neuronx.

I could fix the problem by setting n_positions=4096 before compiling the model.

neuron_model = LlamaForSampling.from_pretrained('./Llama-2-13b-split', batch_size=1, tp_degree=xla_device_count, n_positions=4096, amp='f16')

Maybe this flag should be added to the tutorial to avoid other people having to deal with the same problem :)

from transformers-neuronx.

Thanks @dennj for pointing out the issues. Yes, you can increase the number of positions by setting the n_positions variable. For example, to support up to 4k positions, you could do the following:

neuron_model = LlamaForSampling.from_pretrained(‘./Llama-2-13b-split’, batch_size=1, tp_degree=xla_device_count, amp=‘f16’, n_positions=4096)

We’ll update the documentations accordingly.

from transformers-neuronx.

Thanks :)
Updating the documentation will be helpful.

However the behaviour is non-deterministic, is ok to have a random behaviour in the library?

from transformers-neuronx.

Hi dennj - okay to close this ticket or do you have any other questions?

from transformers-neuronx.

Closing since there were no further comments

from transformers-neuronx.