alexwan0 / infini-gram Goto Github PK

Wondering if someone might be able to help with this error I'm getting when I run the chunked_training.py after downloading the linked pile data ? thanks

$ python chunked_training.py

None of PyTorch, TensorFlow >= 2.0, or Flax have been found. Models won't be available and only tokenizers, configuration and file/data utilities can be used.
Generating train split: 0 examples [00:00, ? examples/s]Failed to read file '/infini-gram/data/pile-val/tokenizer.json' with error <class 'pyarrow.lib.ArrowInvalid'>: JSON parse error: Column() changed from object to string in row 0
Generating train split: 1 examples [00:00, 10.13 examples/s]

raise ValueError(ValueError: Not able to read records in the JSON file at /data/pile-val/tokenizer.json. You should probably indicate the field of the JSON file containing your records. This JSON file contain the following fields: ['version', 'truncation', 'padding', 'added_tokens', 'normalizer', 'pre_tokenizer', 'post_processor', 'decoder', 'model']. Select the correct one and provide it as field='XXX' to the dataset loading method.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/infini-gram/chunked_training.py", line 13, in
dataset = load_dataset(
^^^^^^^^^^^^^

datasets.exceptions.DatasetGenerationError: An error occurred while generating the dataset