Comments (10)
my GPU only has 6G memory which is fine to run the code. dont know your setting
from holy-edge.
Sorry,I forgot to tell you.I use 1080ti GPU and 32GB memory.I can run your code with my own data.But Memory error is happened when the iteration is about 13000.I obverse top command in ubuntu, i find Memory is insufficient(not gpu memory, is computer memory).If you could tell me some suggestion,I really appreciate what you’ve done。
from holy-edge.
Sorry, I have not had such error. This code will save a checkpoint each 100 steps, maybe 13000 is too big in your dataset( I trained at 30000 steps without memory error).
from holy-edge.
I already changed to save a checkpoint with each 1000 steps, but the error is still happen.Did you run this code on windows? Thank you.
from holy-edge.
I run it on Ubuntu 16.04.
from holy-edge.
Could you please give details of your data-set ? It might be the case that your image sizes are quite large and since the VGG base model is full convolutional
the intermediate representations overflow they GPU memory.
from holy-edge.
I also encountered the same problem with you, my server configuration is 8 Tesla p20, memory is 512G. Even with this configuration, memory error occurs after approximately 13,000 iterations during training. Will you solve the problem?@CangHaiQingYue
from holy-edge.
@Jasontachiangwu Well, when I cancelled the 'summary_write' operator, this problem was gone.
So I guess there maybe some bug in 'summary'. You may rebuild your own code.
from holy-edge.
@CangHaiQingYue,After I updated the tensorflow version to r1.8, no problem was found after training. It may be that there is a bug in the summary
.
from holy-edge.
with same code on windows 10 and ubuntu 16.04, tensorflow v1.4.0, 20000 iterations:
memory error occurs in ubuntu 16.04
while no error occurs in windows 10
from holy-edge.
Related Issues (20)
- About Loss Function HOT 1
- cannot clone pre-trained model file HOT 3
- Wrong loss function HOT 2
- Dimension Mismatch HOT 2
- Training does not converge HOT 3
- Error setting up VGG-16 model, Failed to interpret file '/home/xxx/holy-edge/hed/models/vgg16.npy' as a pickle HOT 3
- test error
- This repository is over its data quota. Purchase more data packs to restore access HOT 2
- Could anyone can provide a pretrained model in google drive or baiduyun? HOT 2
- a question about weight decay?
- why the loss changes a little like the picture bllow
- Does anyone have problems with Permission denied when running code with gpu?
- This backport is for Python 2.7 only HOT 1
- How to use the output model with Tensorflow lite for Mobile HOT 1
- Does the optimizer work when the loss is not a tensor?
- Is it true that you didn't train the parameters of VGG16? HOT 2
- How to test and output my own single image using the pretrained model
- Checkpoint can't be downloaded due to data limit reached on git lfs HOT 3
- Reupload the pretrained model?
- Requirements list Tensorflow 2, but code is in Tensorflow 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from holy-edge.