训练效果差，每次回复后面总是要跟一堆重复东西 about alpaca-cot HOT 17 CLOSED

Try training on multiple GPUs, or increase the number of gradient_accumulation_steps.

from alpaca-cot.

这很奇怪，你是单卡训练的吗？

from alpaca-cot.

是的，单卡3090

from alpaca-cot.

已私聊

from alpaca-cot.

I’m encountering the same problem, can you help me?

from alpaca-cot.

will try. But how about tweaking the parameters? do you know what parameter I should adjust to make the response not repetitive and truthful? I tried changing the temperature to 0.8 and setting the greedy decoding to True

from alpaca-cot.

or should I just train it longer by adding the epoch?

from alpaca-cot.

1. Raising the temperature has a slight effect on reducing repetition: higher temperatures have a smoother vocabulary prediction distribution, and a greater chance of picking up words that are not usually picked up.
2. If you set greedy to true, there is no need to configure the temperature.
3. The main reason is that the model is not well trained. More data can be used. If the amount of data is small (50k), it is recommended to train for 3 epoch. If the amount is large (0.5M+), you can only train for 1 epoch.

from alpaca-cot.

by 50k do you mean the number of instructions? my data size is currently 26MB

from alpaca-cot.

yes. 26MB (i mean the number of instructions) data is fairly large. Is the data quality not good?

from alpaca-cot.

well my data size is 26MB and has 50k instructions. The quality should be similar to the alpaca data, as I only translate it to other language using chatgpt.

from alpaca-cot.

I see. This may be due to Bloom's weak ability in your target language. You can collect more data, try larger bloom models, or try the LLM performing well on the target language.

from alpaca-cot.

do you think adding more epochs will solve the problem?

from alpaca-cot.

Yes, it should be better.

from alpaca-cot.

How many epoches do you set? 3 epochs is suitable for 50k instructions.

from alpaca-cot.

currently only one, since I have a limitation on my GPU usage. Three epochs would cost about 10 hours of training (in A100 80GB)

from alpaca-cot.

I'll switch the model to llama and see how it goes

from alpaca-cot.