Corrupted output with llama model about transformers-neuronx HOT 4 CLOSED

Example:
One of my fondest memory is of a weekend in late 1999 when I had come home from a conference. While walking through the house on my wife was surprised me my wife was checking on the fridge was sitting on my eyes were talking to turn into my son, the house) weavinga 7w! I I think of that’0.\nThe\nOne of the end-\nOne One way back then 24 33.\nOne on the whole h d b a day I I I I I I I feel fall trees snowfallfriendfriendfriendss.

from transformers-neuronx.

Hello @dacorvo, The LLaMA model is currently in the "prototype" stage, so such behavior is not surprising.

Prototype (Alpha): An initial in-development version of a model that should be considered a preview of future functionality. A prototype may not be fully functional. A prototype model is not expected to perform well and may also have known accuracy issues. Prototype models may not maintain compatibility across versions.

We'll continue to improve the correctness and performance of these models in future releases. If you are able to provide example prompts and model artifacts (https://github.com/aws-neuron/transformers-neuronx#troubleshooting) that will help us debug and reproduce the issue.

from transformers-neuronx.

@dacorvo,

As a follow-up, could you also let us know what package versions you're using? We've released updates to transformers-neuronx recently with improvements, and the newest version may help.

from transformers-neuronx.

@dacorvo could you please check your version and try the newly released version?

Many thanks

from transformers-neuronx.