Hello! I met "StopIteration" Error when trying to run DNABERT using example data about dnabert HOT 7 CLOSED

Hi,

Thanks for your interest!

This is my first time seeing this issue. Have you successfully installed the apex package?

from dnabert.

When I asked the question, it was not installed.
But, after install, the same error also occurs even add --fp16 flag.
(I installed apex package using conda
"conda install -c conda-forge nvidia-apex")

On my other GPU server, multi-GPU processing works well.
no apex package is installed on that server.
pytorch version is 1.6.0

Although I have downgrade pytorch version to 1.6.0 on the server that error occurred, the same error still occurs.

from dnabert.

Can you try to run the code without the --fp16 tag. If this works, you may install the apex package based on the official instructions (https://github.com/NVIDIA/apex).

from dnabert.

The same error still occurs though run the code without the --fp16 tag.

After upgrade pytorch to 1.8 on the server which has no error for multi-GPU processing,
the same error also occurs.
After install apex package on that server, it still cannot resolve the error.

I guess that the version of pytorch and cudatoolkit may affect parallel processing.
Can you inform the version of packages installed on your workspace?

from dnabert.

I met the same problem with you. Hope it can be solved quickly.

from dnabert.

Hi,

We used pytorch=1.4 + cudatoolkit=10.0 + python=3.6. Could you try this setting?

from dnabert.

It works!!!! 🎉
no error occurs on PyTorch 1.4

This problem seems to be caused by a PyTorch update
https://github.com/huggingface/transformers/issues/8145#issuecomment-721044942
Since, in a few cases, no error occurs on PyTorch >= 1.5, it seems that there is a way to resolve this issue on a higher version of PyTorch.
Unfortunately, I cannot find a way to fix this issue.

A simple way is to downgrade PyTorch to 1.4.

It would be helpful that updating conda install pytorch torchvision cudatoolkit=10.0 -c pytorch to conda install pytorch=1.4 torchvision cudatoolkit=10.0 -c pytorch -c conda-forge on README.md

Thanks for your kind response with interest. :)

from dnabert.