alexwan0 / infini-gram Goto Github PK
View Code? Open in Web Editor NEWAn unofficial implementation of the Infini-gram model proposed by Liu et al. (2024)
License: MIT License
An unofficial implementation of the Infini-gram model proposed by Liu et al. (2024)
License: MIT License
Wondering if someone might be able to help with this error I'm getting when I run the chunked_training.py after downloading the linked pile data ? thanks
$ python chunked_training.py
None of PyTorch, TensorFlow >= 2.0, or Flax have been found. Models won't be available and only tokenizers, configuration and file/data utilities can be used.
Generating train split: 0 examples [00:00, ? examples/s]Failed to read file '/infini-gram/data/pile-val/tokenizer.json' with error <class 'pyarrow.lib.ArrowInvalid'>: JSON parse error: Column() changed from object to string in row 0
Generating train split: 1 examples [00:00, 10.13 examples/s]raise ValueError(ValueError: Not able to read records in the JSON file at /data/pile-val/tokenizer.json. You should probably indicate the field of the JSON file containing your records. This JSON file contain the following fields: ['version', 'truncation', 'padding', 'added_tokens', 'normalizer', 'pre_tokenizer', 'post_processor', 'decoder', 'model']. Select the correct one and provide it as
field='XXX'
to the dataset loading method.The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/infini-gram/chunked_training.py", line 13, in
dataset = load_dataset(
^^^^^^^^^^^^^datasets.exceptions.DatasetGenerationError: An error occurred while generating the dataset
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.