How to run examples/pytorch/token-classification with dataset about lightseq HOT 12 OPEN

Hi ! Thanks for reporting this issue with wikicorpus, we implemented a fix in huggingface/datasets#2844

from lightseq.

Maybe you can try to change this line?

from lightseq.

I already downloaded the dataset to disk using https://github.com/NVIDIA/DeepLearningExamples/blob/master/PyTorch/LanguageModeling/BERT/data/create_datasets_from_start.sh. I am wondering how to use those data ? Thanks !

from lightseq.

I got following error with dataset "wikicorpus"

CMD: python3.7 -m torch.distributed.launch
--nproc_per_node=1
$THIS_DIR/run_ner.py
--model_name_or_path bert-large-uncased
--dataset_name wikicorpus
--dataset_config_name raw_en
--output_dir ./test-ner-no-wikicorpus
--cache_dir ./cache-wikicorpus
--do_train
--do_eval
--num_train_epochs 1

Error message:
Downloading and preparing dataset wikicorpus/raw_en (download: 1.25 GiB, generated: 3.16 GiB, post-processed: Unknown size, total: 4.41 GiB) to ./cache-wikicorpus/wikicorpus/raw_en/0.0.0/8665d716c08f102e87fdbb711326cbdf12c7ce810962819f1c71ca294d722774...
Downloading: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1.35G/1.35G [15:12<00:00, 1.48MB/s]
Traceback (most recent call last):
File "/usr/local/lib/python3.7/dist-packages/datasets/builder.py", line 1103, in _prepare_split
writer.write(example, key)
File "/usr/local/lib/python3.7/dist-packages/datasets/arrow_writer.py", line 342, in write
self.check_duplicate_keys()
File "/usr/local/lib/python3.7/dist-packages/datasets/arrow_writer.py", line 353, in check_duplicate_keys
raise DuplicatedKeysError(key)
datasets.keyhash.DuplicatedKeysError: FAILURE TO GENERATE DATASET !
Found duplicate Key: 519
Keys should be unique and deterministic in nature

During handling of the above exception, another exception occurred:

from lightseq.

It seems a dataset error, not lightseq's error. Anyway, you can try to delete the cache.

from lightseq.

I tried several times, but the error is same, could you have a look ?

from lightseq.

I tried several times, but the error is same, could you have a look ?

You can check huggingface/datasets#2552

from lightseq.

Hi! I run the example:

 sh examples/training/huggingface/run_ner.sh

but get the error:

RuntimeError: CUDA out of memory. Tried to allocate 50.00 MiB (GPU 0; 15.78 GiB total capacity; 7.21 GiB already allocated; 45.75 MiB free; 7.53 GiB reserved in total by PyTorch)
  0%|

How can I solve it?

from lightseq.

when I set per_device_train_batch_size=1, get the error:

 File "/usr/local/python3.7.0/lib/python3.7/site-packages/lightseq/training/ops/pytorch/transformer_encoder_layer.py", line 288, in forward
    assert bs == encoder_padding_mask.size(0) and sl == encoder_padding_mask.size(1)
AssertionError

from lightseq.

This error is caused by the wrong padding mask shape, anyway, batch_size=1 is not usual.
If you encounter GPU out of memory, besides decreasing your batch_size, remember to set smaller max_batch_tokens in lightseq layer config, it will also influence GPU memory usage

from lightseq.

When I set max_batch_tokens=1024 in ls_hf_transformer_encoder_layer.py, I still get the following error:

   f"Batch token numbers {bs * sl} exceeds the limit {self.config.max_batch_tokens}."
ValueError: Batch token numbers 1344 exceeds the limit 1024.

May I change this place?

from lightseq.

from lightseq.