Comments (4)
I just replicated your problem, it is weird that the batch size affects performance so significantly, I suspect that it is due to #1. Yet have no idea why exactly this happens.
from capsnet_guide_pytorch.
emm…I see. I haven't notice the problem mentioned in #1 . Look forward to your fix.
from capsnet_guide_pytorch.
I just pushed an update in the dev branch, you can clone it and see what happens. I have run a few experiments but the issue still exists, and it seems irrelevant to the batch size. Let me summarize the problem (experiments below are all batch_size=64
):
-
If I set
lr=1e-3
as stated in the paper, then the reconstruction loss drops quickly, and finally stays below 30. The reconstructed pictures look good, but the marginal loss does not decline at all and always fixes at 3.644, thus the classifier stays untrained and the accuracy is about 10%. -
If I set
lr=1e-5
, then the marginal loss decreases steadily below 0.01 after a few epochs, and the classification accuracy normally goes above 95%. But the reconstruction loss can maximally decreases to about 50, thus the reconstructed pictures look like this:
-
But in some special cases, even if I set
lr=1e-3
, I can still get both the marginal loss below 0.01 and reconstruction loss below 30, like the sample run in my repo. But that requires a bit of luck and you have to experiment many times to encounter this situation (I'm very lucky to have it in the sample run).
So it seems that the loss surface has more than 2 local minimum and it is tricky to optimize to the global minimum. I have no idea why this happens, maybe I need to recheck every line of code to see if there is any wrong implementation.
from capsnet_guide_pytorch.
I just found that the problem may be caused by the wrong implementation of Decoder
, as I forgot to add ReLU nonlinearity in the network. I have fixed it and it seems everything normal now. Check out if your problem still exists, and feel free to reopen this issue.
from capsnet_guide_pytorch.
Related Issues (2)
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from capsnet_guide_pytorch.