Comments (7)
Hi,
Thanks for your interest!
This is my first time seeing this issue. Have you successfully installed the apex
package?
from dnabert.
When I asked the question, it was not installed.
But, after install, the same error also occurs even add --fp16 flag.
(I installed apex package using conda
"conda install -c conda-forge nvidia-apex")
On my other GPU server, multi-GPU processing works well.
no apex package is installed on that server.
pytorch version is 1.6.0
Although I have downgrade pytorch version to 1.6.0 on the server that error occurred, the same error still occurs.
from dnabert.
Can you try to run the code without the --fp16
tag. If this works, you may install the apex
package based on the official instructions (https://github.com/NVIDIA/apex).
from dnabert.
The same error still occurs though run the code without the --fp16 tag.
After upgrade pytorch to 1.8 on the server which has no error for multi-GPU processing,
the same error also occurs.
After install apex package on that server, it still cannot resolve the error.
I guess that the version of pytorch and cudatoolkit may affect parallel processing.
Can you inform the version of packages installed on your workspace?
from dnabert.
I met the same problem with you. Hope it can be solved quickly.
from dnabert.
Hi,
We used pytorch=1.4
+ cudatoolkit=10.0
+ python=3.6
. Could you try this setting?
from dnabert.
It works!!!! 🎉
no error occurs on PyTorch 1.4
This problem seems to be caused by a PyTorch update
https://github.com/huggingface/transformers/issues/8145#issuecomment-721044942
Since, in a few cases, no error occurs on PyTorch >= 1.5, it seems that there is a way to resolve this issue on a higher version of PyTorch.
Unfortunately, I cannot find a way to fix this issue.
A simple way is to downgrade PyTorch to 1.4.
It would be helpful that updating conda install pytorch torchvision cudatoolkit=10.0 -c pytorch
to conda install pytorch=1.4 torchvision cudatoolkit=10.0 -c pytorch -c conda-forge
on README.md
Thanks for your kind response with interest. :)
from dnabert.
Related Issues (20)
- How can I create my own processor? HOT 1
- There is a bug about attention mask in source code
- Importing error of Transformers HOT 4
- How to get the high attention regions of a given sequence.
- AssertionError in kmer2seq for motif search
- attention maps generated in pre-training stage or fine-turning stage
- Pretraining error
- benchmark for the time and computation cost during the fine-tuning
- Shape of atten.npy
- install packages using pip HOT 2
- the seq longer than 512
- Unable to get motif image HOT 3
- Request for Detailed Information on Training the Tokenizer HOT 1
- Release pretraining data?
- what is the masking ratio
- Installation Issues.
- How to divide our own dataset into test, dev and train data and assign them labels for fine tuning process HOT 2
- Changing max_seq_length does not update max_length in config.json
- early_stop not being triggered?
- How Can I track model loss and accuracy of each epoch during fine-tuning, to make sure model is stable?
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from dnabert.