Comments (5)
The underlying convolution routines won't get any faster when the batch_size goes from 64 -> 128, so it isn't surprising that the overall training doesn't either.
from benchmarks.
@zhaoerchao @ekelsen These benchmark programs are giving for example say 570images/sec where as when you run the same model normally it gives half of that of the benchmark programs gave, why so?
from benchmarks.
@cryptox31 Do you run the program on the same GPU with the same version TF?
from benchmarks.
@zhaoerchao I am currently using 4 Tesla P100 GPUs and running Tensorflow 1.01 inception v3 model, and I am not getting optimum results.
from benchmarks.
Increasing the batch-size will not always increase performance. I am far from an expert. From my testing, I find that each model and hardware combination will have a point where even if there is more memory, increasing batch size is does not help. Increasing batch size normally helps when the step time is very fast and increasing the batch-size slows down the step time enough to hide the transfer times and other calculations that are impossible to "hide" with a very fast step time. I know that is not a very technical explanation. One good example of this is notice that "everyone" runs alexnet at a batch size of 512 or more now but use to run much small batches. I have not been working with ML very long but if you test alexnet with 32, 128, 256, and then 512 on most ML platforms you will see a significant speedup as the batch-size increases. If I remember correctly, even more so on multi-GPU.
Finally, the goal is normally to converge at the best possible top_1. I know people are training with large total batches for ResNet but I have not seen anyone training with 128 per GPU. Of course there is so much happening it likely has happened and I did not see it.
Closing as this is kind of expected. If you are having unexpected results with batch-size 64 or 32 please let me know and I will see if I can figure it out.
from benchmarks.
Related Issues (20)
- How do I get TF2.0 benchmark. HOT 1
- Impossible to test perfzero without docker HOT 2
- not using GCE instance HOT 2
- NotFoundError: No CPU devices are available in this process HOT 1
- Failure when running models with tf_cnn_benchmarks (threadpool in preprocessing.py) HOT 1
- ImportError: cannot import name multi_device_iterator_ops HOT 1
- OP_REQUIRES failed at save_restore_v2_ops.cc:205 : Not found: Key grouping/TCN/res_0_1/layer_normalization_1/beta not found in checkpoint
- TypeError: visualize_boxes_and_labels_on_image_array() missing 1 required positional argument: 'category_index'
- The accuracy of the program running by horovod is low
- resnet50 --use_fp16 error: cuDNN launch failure : input shape ([128,112,112,64])
- Perfzero support for Openshift on RHEL
- How to evaluate worker performance independently on a distributed training
- Alternative/current state of tf_cnn_benchmark HOT 3
- perfzero resnet benchmark is outdated HOT 3
- PerfZero Dataset for RetinaNet is not Avaiable HOT 1
- 4090 multi gpu not support
- What to use as replacement for tf_cnn_benchmark in the official tensorflow models HOT 2
- Docker 22.04 ubuntu and 12.1 cuda? HOT 1
- VariableV1 does not exist in /tensorflow/python/ops/variables.py HOT 2
- ImportError: cannot import name 'variable_v1' from 'tensorflow.python.ops' HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from benchmarks.