Error when runinng script pretrain_gpt2_distributed.sh about megatron-lm HOT 5 CLOSED

I also got this error and found that it was caused when using the default parameters of train/val/test split of 949,50,1 (in the bash script).
As the test set only contains a single example when using the default wikipedia dataset, the np array is returned as a non iterable instead of an iterable of a single item. The accumulate function in line 489 of datasets.py is returning the mentioned error.
changing to 939,40,11 solved this issue

from megatron-lm.

It seems your datasets are empty and the actual error is:

TypeError: iteration over a 0-d array

Could you check the paths and make sure you can create a dataset instance?

from megatron-lm.

Also I have similare error when I run pretrain_gpt2_model_parallel.sh
Both script using python -m torch.distributed.launch

from megatron-lm.

Marking as stale. No activity in 60 days. Remove stale label or comment or this will be closed in 7 days.

from megatron-lm.

Closing - error explained by @PyxAI

from megatron-lm.