`No module named 'datasets'` in `data_prep/book/` about redpajama-data HOT 5 CLOSED

Thanks! I must have mistyped it.

There have been a couple of times I have had to add system dependencies and copy libs from system python to my pyenv python to get things to work as well in various project folders. For this I had to do:

On Ubuntu:

sudo apt install lzma liblzma-dev libbz2-dev

cp /usr/lib/python3.10/lib-dynload/_bz2.cpython-310-x86_64-linux-gnu.so ~/.pyenv/versions/3.10.6/lib/python3.10/lib-dynload/
cp /usr/lib/python3.10/lib-dynload/_lzma.cpython-310-x86_64-linux-gnu.so ~/.pyenv/versions/3.10.6/lib/python3.10/lib-dynload/

I believe the bz2 was also needed in another dataset folder.

For stack exchange I needed:

sudo apt install p7zip-full

For C4 dataset I needed:

sudo apt install git-lfs

These were not documented in the READMEs.

from redpajama-data.

This can be searchable so I'm closing.

from redpajama-data.

Thanks for bringing this to our attention! The error your are observing is due to the datasets library not being installed. We will add this to the dependencies. In the meantime, you can fix the error by running

pip install datasets

in your python environment.

from redpajama-data.

Just discovered that the data_prep/book/ download uses $HOME/.cache/huggingface/ for downloads. My system doesn't have extra space for the home directory to be doing that kind of work so I've symbolically linked it to an external drive.

My main AI machine isn't finished being setup yet so I'm prepping data outside of it.

from redpajama-data.

I should also mention here that the pile download would always fail at or before 40% completion. So to download the data I used wget:

wget -c https://the-eye.eu/public/AI/pile_preliminary_components/books3.tar.gz

from redpajama-data.