Comments (5)
Hi @ggsonic
That's an interesting observation. We primarily use the timm library to facilitate training, but did not notice this issue with dataloader.
Would you please provide more information regarding your setup which is used for training ?
Best,
from gcvit.
CPU:AuthenticAMD AMD EPYC 7763 64-Core Processor . GPU: A100. OS: Ubuntu 20.04 CUDA: cuda_11.3 . pytorch: 1.10.2+cu113 . timm 0.5.4.
we are using SLURM . 8 A100 GPU , each task with num_workers =8 . you can see in the snapshot only 8 cpu processes 100% working(we are using 8 A100 GPU not 4 as in your paper ).
from gcvit.
Hi @ggsonic
Thank you for sharing this information. We trained the model using 4 computational nodes, as specified in the paper. However, each node comprises of 8 GPUs, hence a total of 32 GPUs for this task. Using batch size of 128, training finished in around 22 hours on NVIDIA's NGC cluster.
According to your snapshot, I believe the issue CPU is clearly the bottleneck-- the A100 GPUs are consuming data in a higher rate and the CPU cores have a hard time to catch up. The timm library, which is primarily used by our work, has done a great job to address these bottlenecks in the dataloader, but it is still challenging for blazing fast hardware such as A100.
Best,
from gcvit.
it was my fault. i changed the code for using slurm,and set wrong create_loader params. Then i set distributed=True
and everything is ok now. Thanks!
from gcvit.
Hi @ggsonic ,
Thanks for letting us know. In addition, we have updated our model and provide checkpoint with improved performance.
Best
from gcvit.
Related Issues (20)
- Global query: wrong input transformation HOT 1
- is there a large release, please ? HOT 1
- image input for channel changing HOT 3
- Reliable comparison with Swin: fixed window size HOT 1
- training hparam clarifications HOT 3
- x = x.reshape(B, 1, self.N, self.num_heads, self.dim_head).permute(0, 1, 3, 2, 4) reshape error!!!! HOT 3
- New third part implementation HOT 2
- Using timm gcvit models and getting errors when using resolution other than 224x224 HOT 2
- Backbone Feature Maps for One-Stage Detectors HOT 5
- Validate using public checkpoint, Accuracy extremely low HOT 4
- Is there any documentation to train a semantic segmentation model using a custom dataset? HOT 3
- Hi, Where can I download GC_ViT-B_384 pretrained weight ? HOT 6
- Third-Party JAX/Flax implementation HOT 2
- How to use "--input-size" properly? HOT 2
- Getting an error for the output shape for each stage in GC ViT
- Getting an error for the output shape for each stage in GC ViT HOT 1
- Shape errors in data loading when using train.py HOT 1
- Could anyone help me with the data loading process? HOT 1
- How to Visualize the global attention, as shown in Figure 1? HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from gcvit.