Document use of Mistral about maxtext HOT 6 CLOSED

It should work on a v3-8.
You can also try the decode.py function but for me it worked on the 7b models (gemma or mistral).

from maxtext.

Amazing @borisdayma! We don't actually official support Mistral (we do support Llama and Gemma) but we're thrilled things are working for you!

from maxtext.

Looks like this is actually available: https://github.com/google/maxtext/blob/main/end_to_end/test_mistral.sh

The only thing I had to do was replace tokenizer.mistral with tokenizer.model (is it a typo or did you rename it in your bucket?).
Also I chose to convert the bfloat16 weights to float32 instead of float16 which I think could bring some imprecision.

from maxtext.

Can I ask what kind of TPU are you using for the test, @borisdayma? I do have available a v4-32 that I'd like to use to do continue pre-training on Llama2/Mistral 7B, but other frameworks seem sub-optimal so far to me.

from maxtext.

Yeah your inference test of mistral is correct. I compared with transformers output and was getting the same.

from maxtext.

I'm closing this issue because Mistral seems to already work well after further testing.

from maxtext.