Comments (8)
I saw that some time ago. Neither TF nor Theano have such ops. And I am not that interested in delving deep into the guts of TF or Theano.
I had my friend ask Prof. Killian Weinberger for a temporary measure to reduce memory consumption in DenseNets, and the answer was using the BC architecture with lesser width (at the cost of performance obviously). That seems to be the way to go for now.
I'll keep the issue up, in case some time in the future these frameworks will implement such ops.
from densenet.
It’s not exactly the same as the original but if you check the current version by gpleiss he’s altered it to gradient checkpointing like this.
from densenet.
Sounds good, I created https://stackoverflow.com/questions/46124965/tensorflow-shared-memory-allocation-of-recursive-features-for-densenet to see if anyone has advice.
from densenet.
Hey @titu1994 I used your code and someone elses to show how to do a memory efficient version in tensorflow:
https://github.com/joeyearsley/efficient_densenet_tensorflow
I'm not exactly sure how to do in Theano or CNTK but guessing it could be merged in with an explicit warning that the efficient version only works with the tf backend?
Or maybe someone could help make a gradient checkpointing end point in Keras such that it calls version specific ones.
Sorry @ahundt as you are probably being spammed by these notifications atm.
from densenet.
This is definitely very useful, but it is not technically the same technique applied by the memory efficient DenseNets in the original Torch codebase.
Their, they cache the activation of BN and Relu of each of the layers so that multiple calls to those layers use precomputed values (as far as I remember).
In either case, gradient checkpointing is also a useful tool to train extremely large DenseNets.
from densenet.
Oh, that's interesting. Then I suggest you send a PR to this repository to be properly credited (f you have the time). If not, then I can manage to do it in a week or two.
from densenet.
The problem is the Keras backend, I can do a TF only one which raises and issue otherwise?
from densenet.
I'm thinking of adding a separate file which follows most of what was done in the original densenet.py
script, but uses the TF only functions.
Maybe something like densenet_efficient.py
or something.
from densenet.
Related Issues (20)
- Cifar10 weights HOT 1
- no longer works with newest keras HOT 8
- About implementation of __dense_block HOT 2
- Running Densnet in CPU HOT 1
- Why reshape in fc-densenet in top layer before applying softmax? HOT 2
- AttributeError: 'NoneType' object has no attribute 'get_file' HOT 2
- Where is connection to 12 layers of each dense block? HOT 1
- About DenseNet HOT 1
- About Densenet architecture HOT 1
- Plans for memory efficient implementation in Keras? HOT 1
- How to upload my own dataset instead of the Cifar10 dataset HOT 1
- How to use 'DENSENET_121_WEIGHTS_PATH_NO_TOP'? HOT 1
- Mistake in L2 regularization HOT 1
- creat model HOT 2
- creat model HOT 1
- imagenet_inference
- inter_channel HOT 1
- Poor CIFAR100 accuracy HOT 1
- Error about normalize_data_format
- Unable to convert to frozen graph or checkpoint
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from densenet.