Comments (18)
@ssnl @xingyizhou Does this bug still exist with pytorch >= 1.0?
from pytorch-pose-hg-3d.
Hi,
I have investigated this problem (on another project, while I can not reproduce the bug on this project). It seems it is caused by very large intermediate features (e.g. > 10000) before batch normalization. Then the train() mode is on, it will be normalized be itself so training is OK. But when eval() mode is on, a slight difference (of the intermediate feature) with the BN mean/std from training will results in large offsets for output. I don't know the causal of the problem but it looks mathematically reasonable. However, down-grading PyTorch version to 0.1.12 will eliminate the problem. Please notify me if you have any other observations on this bug. Thanks!
from pytorch-pose-hg-3d.
Hi all,
As pointed by @leoxiaobin, turn off cudnn of BN layer resolves the issue. It can be realized by set torch.backends.cudnn.enabled = False
in main.py
, which disables cudnn for all layers and slows down the training by about 1.5x time, or re-build pytorch from source by hacking cudnn in BN layers https://github.com/pytorch/pytorch/blob/e8536c08a16b533fe0a9d645dd4255513f9f4fdd/aten/src/ATen/native/Normalization.cpp#L46 .
from pytorch-pose-hg-3d.
Hi,
As far as I know, it should be caused by a pytorch internal bug in BN. You can comment model.eval()
in testing to see if the validation acc gets better (but it still won't match the desired performance). The bug should not be reproducible. And re-train the network once more (better on another machine) should have different results. Or you can downgrade your pytorch version below 0.1.12, which is a version where I haven't met/ heard about this bug (but still not guaranteed). Please let me know if the above solutions help. Thanks!
from pytorch-pose-hg-3d.
I tried for another 3 times. The train acc is approximately 0.87 during the last epoch(the 60th epoch) but the validation acc changes every time and always lower than 0.50. The validation acc is around 0.80 in the 55th epoch so it seems that there is a sudden drop during the last epoch and I notice that the training loss gets slightly higher during the last epoch.
from pytorch-pose-hg-3d.
Hi,
Thanks for reporting the problem. However I don't have other solutions yet and will keep looking into it. It might not be a bug of the code, since an isolated implementation of HourglassNet (I am not sure if the bug is from the network architecture) also has this problem (bearpaw/pytorch-pose#33). People there suggest using learning rate 1e-4. You can have a try to see if the bug still exists.
from pytorch-pose-hg-3d.
Hi,
Thanks for your advice. Yes it works if using LR 1e-4. The val acc is 0.80+ in this way.
from pytorch-pose-hg-3d.
Hi,
Yes I think it is reasonable. Sure I will notify you if I observe something new. Thanks for your reply!
from pytorch-pose-hg-3d.
IIRC, your repo sets batch size to 1. If that is the case it's not really a PyTorch bug. Running stats with batch size = 1 is unstable itself.
from pytorch-pose-hg-3d.
Thanks for the suggestion! The training batch size is 6 and testing is 1. When testing, eval() mode is on and the batch size does not affect the computation.
from pytorch-pose-hg-3d.
from pytorch-pose-hg-3d.
Get it. Thanks.
from pytorch-pose-hg-3d.
Oh I still want this issue to be opened to wait for better solutions...
from pytorch-pose-hg-3d.
Sure! My bad.
from pytorch-pose-hg-3d.
@wangg12 I am doing experiments to observe if the bug exists in pytorch >= 1.0.
from pytorch-pose-hg-3d.
@ wangg12 我正在做实验,以观察pytorch> = 1.0中是否存在该错误。
Can you meet this error when the version of pytorch >= 1.0
from pytorch-pose-hg-3d.
@ujsyehao 你好,请问你的实验结果如何?
from pytorch-pose-hg-3d.
Hi all,
As pointed by @leoxiaobin, turn off cudnn of BN layer resolves the issue. It can be realized by settorch.backends.cudnn.enabled = False
inmain.py
, which disables cudnn for all layers and slows down the training by about 1.5x time, or re-build pytorch from source by hacking cudnn in BN layers https://github.com/pytorch/pytorch/blob/e8536c08a16b533fe0a9d645dd4255513f9f4fdd/aten/src/ATen/native/Normalization.cpp#L46 .
torch.backends.cudnn.enabled = Falsein
main.py`
Should it be "torch.backends.cudnn.benchmark = False"?
If I have followed this step, I need not modify main.py, right? :
For other pytorch version, you can manually open torch/nn/functional.py and find the line with torch.batch_norm and replace the torch.backends.cudnn.enabled with False
from pytorch-pose-hg-3d.
Related Issues (20)
- s = min(s, max(img.shape[0], img.shape[1])) * 1.0 AttributeError: 'NoneType' object has no attribute 'shape' HOT 4
- Confidence of Detected Joints
- The defination in dataset file
- h36m benchmark result for categories
- Stage1: 2D pre-trained model trained to epoch=0?
- about the permission of pre-processed Human3.6M dataset
- about the pre-processed of Human3.6M
- RUN ERROR
- Performance On MPI-INF Dataset
- The preprocess of mpii is incorrect when input_h!=input_w
- The src point is wrong in image.py
- some problems occur running demo HOT 2
- the model behave worse when detect the squat
- download the models HOT 2
- 段错误,核心已转储 HOT 1
- how to using the camera of my own pc??
- root joint
- I solved 。。。
- Is it possible to input 2D coordinates obtained by other methods to obtain 3D coordinates in your model?
- demo has error HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from pytorch-pose-hg-3d.