Comments (11)
To enable multi-GPU training, you will need to change this line to MultiGPUTrainer.
Expect some adventures when using multi-GPU for this project. I am not sure about the behavior.
from generative_inpainting.
@RyanHTR I changed this line to MultiGPUTrainer. But I got an error "TypeError: 'NoneType' object is not callable" which I can't figure it out. Do you have this problem?
from generative_inpainting.
@RyanHTR Hello, RyanHTR, can you train the network successfully on multi-GPU?
from generative_inpainting.
@JiahuiYu There is a bug for 'NoneType object is not callable' None()
from generative_inpainting.
@1900zyh This is not bug. Loss should be None for multi-GPU training.
from generative_inpainting.
@JiahuiYu I think it should be
assert loss is None, 'For multigpu training, graph_def should be provided, instead of loss.'
Or it will report TypeError
from generative_inpainting.
@1900zyh Ohhhh I see. Thank you!
from generative_inpainting.
I have 4 GTX 1080Ti GPUs and each gpu can handle batch size of 16 that means if I use all the gpus I can change batch size to 64. But when I do that my GPUs ran out of memory.
Am assuming here that ng.train.MultiGPUTrainer
uses data parallelism to split input data (64 batch size) in to 4 gpus where each gpu gets 16 batch of images.
Because of that Issue I can only train on batch size of 16, whether I use 4 gpus or 1 gpu.
What are your thoughts about this?
from generative_inpainting.
@bis-carbon The batch size here is the per-gpu batch size.
from generative_inpainting.
Thank you for your quick response and great work.
from generative_inpainting.
@1900zyh @bis-carbon @lipanpeng Hi. Have you figured out the issues that how to use multi gpu for training. If so, kinldy let me know, I am struggling. Thanks in advance
from generative_inpainting.
Related Issues (20)
- not able to train the model
- Training of dataset
- required broadcastable shapes [Op:Mul]
- How to change the learned model
- Terrible result image
- NotImplementedError: Cannot convert a symbolic Tensor HOT 1
- Increase/decrease of input/output dimensions
- New easy to use symmetric face inpainting
- URGENT
- I don't have access to the Google download Pretrained models HOT 2
- Training on own dataset HOT 1
- form of flist HOT 1
- I write something easy to modify. HOT 2
- Why remove l1_loss in v2 ?
- Why split in 2 instead of 3 in gen_conv ?
- two same repo?
- AssertionError loss_value is NaN
- Implementation Discrepency Relative to Publication
- Trouble with quality of results using pretrained model
- NVIDIA Issue and Modifying Variables in Training
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from generative_inpainting.