Crash when training Llama 2 from HF on ROCm about composer HOT 3 CLOSED

Hey @rlrs, this error usually occurs when there is a token index that is larger than the embedding matrix of the model. I suspect you tokenized your data with a different tokenizer than the model you are now trying to train. Please try making sure you run your pretokenization with the same tokenizer as the model you are training.

from composer.

Hey @rlrs, this error usually occurs when there is a token index that is larger than the embedding matrix of the model. I suspect you tokenized your data with a different tokenizer than the model you are now trying to train. Please try making sure you run your pretokenization with the same tokenizer as the model you are training.

Brilliant! You likely saved me many hours of trying to figure this out. For some reason I thought the dataset was being tokenized on the fly.

As for the missing stack trace and difficulty debugging, I assume that is an upstream issue. It would have been a lot easier to figure out with a decent error message. I will make a Pytorch issue on that.

from composer.

Hm... I don't think we've seen this error before. I will ask around internally with those more familiar with running on AMD.

I would guess this is a torch thing, and maybe it's worth trying not nightly...? Shot in the dark.

from composer.