Comments (11)
This error does occur sometimes at higher ratios with your resizing code. I had just assumed that my particular case (about 10GB of video memory free) was not perfectly protected by the resize, which was perhaps written for the 12GB commonly available. If I manually resize the images down myself to a lower resolution than your code does I can usually get the ratio that failed to work (e.g. 1.0)
I will see if I can resolve that particular error by leaving out CUDNN (or perhaps using a different version, I am on 5.1).
But the larger issue for me is breaking through the limitation that requires the resizing in the first place. To put my question more simply, why do you have to resize the images down, and is there a way to increase that maximum resolution? Though the final results are upsized in your code as a last step, they only have the detail and information of at most a 700x500 image. I would like to run 1000x1000 and larger source and style images through the system at full resolution for even better results.
from deep-image-analogy.
Update! I built Caffe without CUDNN and was able to get higher resolutions. Also, I get a more sensible error that I am out of memory (and can see the memory approach the limit using nvidia-smi). I still suspect the CUDNN error above was about memory, as it occurred only with increased image size and/or scale. It seems that CUDNN uses additional memory in this case.
By raising the resize limits in your code I was able to work with content and style images close to 1000x1000 up to a ratio of .7. With CUDNN I was only able to get up to ratio .3 with the same images. I'll try testing with different versions of CUDNN (I used 5.1) to see if any have better memory performance. The performance with CUDNN was faster, but not substantially so (29 seconds vs 34 without in one case) -- not enough to give up the higher resolution possibilities.
I'd still love to be able to go higher than this improved resolution. I wonder if changing out VGG-19 for a less resource intensive trained net would help (as it does with Neural Style and other style transfer systems). I tried replacing it with VGG-16 to test but it did not work. I guess there are more code changes required to utilize a different trained net.
from deep-image-analogy.
Yeah, you are right. CUDNN exactly uses additional memory so it may cause such error when the size become larger.
I think there are two ways to increase the resolution.
- You can use a better GPU with more memory which can handle images of larger size.
- You can try to reduce the number of threads in a block (20x20 for now) which could reduce the memory it needs for computing.
from deep-image-analogy.
In our project, we would resize an image if it's too large. So if you input a 700x700 image, the program will resize it. Did you get this error using our resize codes or without those codes?
Because this error may not be definitely caused by out of memory.
from deep-image-analogy.
The reason could be cuDNN, you can check a similar issue here BVLC/caffe#2197.
Actually our project does not need cuDNN so you can get rid of it.
from deep-image-analogy.
I tried reducing the threads in a block, but got no noticeable reduction in memory usage. Perhaps I changed it incorrectly. I edited only DeepAnalogy.cu
I changed the 6 lines like:
dim3 threadsPerBlockAB(20, 20, 1);
to
dim3 threadsPerBlockAB(10, 10, 1);
This did not get me any noticeable memory reduction.
I also then changed the surrounding lines like:
dim3 blocksPerGridAB(data_A_size[curr_layer].width / 20 + 1, data_A_size[curr_layer].height / 20 + 1, 1);
I replaced the 20 with a 10 in those lines, in case that was necessary.
Again, no noticeable memory reduction.
As far as the better GPU, I can get occasional access to a P5000 with 16GB of memory, which will certainly help.
Thanks for your help -- I am getting some stunning results.
from deep-image-analogy.
I don't believe that you are running out of memory.
I have a GT 740 with 4GB of memory and compiled the code with <UseCuDNN>true</UseCuDNN>
, and the generated binary runs without error. I've used input images up to 30 megapixels with a ratio of 1, which I am assuming is no problem for the program since the input images are resized after the parameters are set. It seems that the minimum area of the input image is 40000 pixels^2 with the image being at least 200x200 pixels in height x width, and the maximum area probably being 350464 pixels^2 with the maximum height or width being 700 pixels. Maybe the aspect ratio of the images need to be in a range where the height and width can be equal to or greater than 200 and less than or equal to 700 while the area is kept between 40000 pixels^2 and 350464 pixels^2.
The source code for resizing the input images is in lines 104 through 229 in DeepAnalogy.cu.
from deep-image-analogy.
same problem,
``F0603 15:33:11.045892 38 syncedmem.cpp:56] Check failed: error == cudaSuccess (2 vs. 0) out of memory
*** Check failure stack trace: ***
@ 0x7f18bb45ddaa (unknown)
@ 0x7f18bb45dce4 (unknown)
@ 0x7f18bb45d6e6 (unknown)
@ 0x7f18bb460687 (unknown)
@ 0x7f18bb8cdb61 caffe::SyncedMemory::to_gpu()
@ 0x7f18bb8ccef9 caffe::SyncedMemory::mutable_gpu_data()
@ 0x7f18bb8ced43 caffe::Blob<>::mutable_gpu_diff()
@ 0x7f18bb989943 caffe::PoolingLayer<>::Backward_gpu()
@ 0x7f18bb94b5f7 caffe::Net<>::BackwardFromTo()
@ 0x40af70 my_cost_function::f_gradf()
@ 0x42d045 lbfgs::gpu_lbfgs()
@ 0x42ccef lbfgs::minimize()
@ 0x40b539 deconv()h
@ 0x42543c DeepAnalogy::ComputeAnn()
@ 0x42108f main
@ 0x7f18ba25ff45 (unknown)
@ 0x4075c9 (unknown)
@ (nil) (unknown)
using the prameters like this:
./demo deep_image_analogy/models/ deep_image_analogy/demo/img_analogy/content/246_2.png deep_image_analogy/demo/img_analogy/align/5467_00000.png deep_image_analogy/demo/output/ 0 0.9 2 1
but ratio .5 is ok
the memory is like the following:
+------------------------------------------------------+
| NVIDIA-SMI 352.63 Driver Version: 352.63 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 GeForce GTX 980 Ti Off | 0000:03:00.0 Off | N/A |
| 45% 84C P2 182W / 250W | 5804MiB / 6142MiB | 99% Default |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 10464 C python 1009MiB |
| 0 10646 C ./demo 3768MiB |
why the model take so much memory?
from deep-image-analogy.
Actually the feature maps and the VGG model take much memory.
from deep-image-analogy.
@jpcreamer what changes did you have to make in VS to get the deep_image_analogy project to build without CUDA? I was successfully able to build Caffe with CPU only via CommonSettings.props
(and also in the one coped to deep_image_analogy), but the build trace still shows that deep_image_analogy is still trying to use nvcc to perform the build.
from deep-image-analogy.
Hi everyone
Im so much fascinated by this technology, after 22 years in computer graphics for films and TV, i feel as motived as in my early start :)
Now i don't have enough knowledge and experience, maybe neither the right side of the brain too :) to fully be confortable enought with all of this to build a new version of the exe which will allow me to raise the resolution, to be able to choose it by a parameters ideally...
For now, i'm on a modern 8 core fast CPu, 64gig ram which i may upgrade to 64 and a 11G GPU Nvidia GTX 1080x GTI.
I was wondering if a gentleman or a smart lady would be gracefull enough to help me and compile and build a new exe which could bypass this very low resolution..
I would be immensively thanksfull if by chance someone could do this for me and i imagine for the joy of all Github's community
Many thanks
vincent
[email protected]
http://fr.linkedin.com/in/vincentthomas
from deep-image-analogy.
Related Issues (20)
- lcaffe
- ./demo: error while loading shared libraries: libcaffe.so.1.0.0-rc3: cannot open shared object file: No such file or directory
- demo keeps running HOT 1
- next_layer = curr_layer + 2 during deconv, but it is + 1 in the paper HOT 1
- Code inconsistency from the Paper HOT 5
- Demo File Location HOT 4
- undefined reference to symbol '_ZN6google21ParseCommandLineFlagsEPiPPPcb'
- Multiple commands HOT 1
- undefined reference to ‘‘caffe::Net<float>::Net(...)" while sh scripts/make_deep_image_analogy.sh HOT 1
- For the preinstalled caffe,how to run your demo?
- Demo run with large ratio causes hard crash
- Is it possible to use multiple GPU to speed it up ?
- libgdk-x11-2.0.so.0: undefined reference to `XRRFreeMonitors' HOT 3
- linux : demo crashing free(): invalid pointer: 0x0000000002281d10 HOT 1
- Minimal GPU requirements for demo HOT 1
- The resultAB.PNG resultBA.png not change,and flowAB.txt flowBA.txt data are all 0,what's happen?
- which version of protoc can be used in this project?
- What are the visual attributes considered here? HOT 1
- Question concerning Build
- Output correspondence map
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from deep-image-analogy.