Comments (6)
Hi 2 * P40 probably are not enough for the task semantic segmentation for example 2*P40 even can’t run the Deeplab v3+ or DANet with ResNet-101.
Following is the minimum resource to run SETR (bs=8) on Cityscapes you can see it is on par with most existing segmentation models.
SETR-Naive-DeiT, 8 * 11.5G
SETR-PUP-DeiT, 8 * 12.8G
SETR-MLA-DeiT, 8 * 12.1G
from setr.
I tried to train SETR-Naive-DeiT&SETR-MLA-DeiT on 4* TITAN RTX 24g GPU, I set samples_per_gpu=1 on config/SETR/.py,so my batch size is 4. But I can not start training cus OOM. You said SETR-Naive-DeiT, 811.5G SETR-PUP-DeiT, 812.8G SETR-MLA-DeiT, 812.1G . But It is different from my experimental result. How could I do to reduce the memory used?
from setr.
we don't have such a problem from our side
from setr.
RuntimeError: CUDA out of memory. Tried to allocate 44.00 MiB (GPU 6; 23.70 GiB total capacity; 21.91 GiB already allocated; 36.81 MiB free; 22.28 GiB reserved in total by PyTorch)
return nll_loss(log_softmax(input, 1), target, weight, None, ignore_index, None, reduction)
File "/home/ls/anaconda3/envs/open-mmlab/lib/python3.8/site-packages/torch/nn/functional.py", line 1605, in log_softmax
ret = input.log_softmax(dim)
RuntimeError: CUDA out of memory. Tried to allocate 44.00 MiB (GPU 0; 23.70 GiB total capacity; 21.91 GiB already allocated; 36.81 MiB free; 22.28 GiB reserved in total by PyTorch)
Traceback (most recent call last):
File "/home/ls/anaconda3/envs/open-mmlab/lib/python3.8/runpy.py", line 194, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/home/ls/anaconda3/envs/open-mmlab/lib/python3.8/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/home/ls/anaconda3/envs/open-mmlab/lib/python3.8/site-packages/torch/distributed/launch.py", line 260, in
main()
File "/home/ls/anaconda3/envs/open-mmlab/lib/python3.8/site-packages/torch/distributed/launch.py", line 255, in main
raise subprocess.CalledProcessError(returncode=process.returncode,
subprocess.CalledProcessError: Command '['/home/ls/anaconda3/envs/open-mmlab/bin/python', '-u', './tools/train.py', '--local_rank=7', 'configs/SETR/SETR_PUP_768x768_40k_cityscapes_bs_8.py', '--launcher', 'pytorch']' returned non-zero exit status 1.
When i tried to train your model "SETR_PUP" by the "./tools/dist_train.sh configs/SETR/SETR_PUP_768x768_40k_cityscapes_bs_8.py 8", i get the above issues even my machine has 8*3090 with 24G. Can you help me to solve it? Thank you.
from setr.
try following three variants with DeiT
SETR-Naive-DeiT, 8 * 11.5G
SETR-PUP-DeiT, 8 * 12.8G
SETR-MLA-DeiT, 8 * 12.1G
from setr.
I have a similar issue when training on my own dataset. Always it is CUDA out of memory. I am using 6 GPUs with 12GB (4 GTX 1080TI, 2 RTX 2080TI). Is there any way to train without getting error?
from setr.
Related Issues (20)
- AttributeError: 'DataContainer' object has no attribute 'shape' HOT 1
- error for using dist_train.sh HOT 2
- typo in Dockerfile HOT 1
- mmcv error ,python3.7.0,pytorch 1.9.1,windows HOT 2
- about single GPU HOT 1
- could you share the demo of SETR? HOT 1
- SETR-Naive-Base model HOT 5
- ZeroDivisionError: integer division or modulo by zero HOT 1
- Questions about "cls_token" and "pos_embed" in the code HOT 10
- multi-scale testing
- Difference with ViT
- 单GPU问题
- RecursionError while training the custom dataset
- ImportError: cannot import name 'container_abcs' from 'torch._six'
- RuntimeError: cuDNN error: CUDNN_STATUS_EXECUTION_FAILED
- Batch size modification HOT 3
- 安装环境问题 HOT 1
- timm issue HOT 2
- RuntimeError: no valid convolution algorithms available in CuDNN
- AttributeError: module 'signal' has no attribute 'SIGKILL'
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from setr.