Comments (17)
Lower the learning rate.
from mnc.
@hgaiser Thanks ! Besides,
I want to train from scratch without finetuned model, when I comment the "weights-" line in mnc_5stage.sh, why the the training data becomes the voc2007? I just want to train my own dataset and put it all in the VOCdevkitSDS folder
from mnc.
Why do you want to train from scratch? My advice is to use the pretrained model actually.
from mnc.
@hgaiser hi~ when i change the input data size to 1_3_1200*1200, it happend to this error and the loss become NaN. i want to know why it happends?and low the learning rate could be work?
from mnc.
Not sure what you mean with that input size. Images are by default resized such that the shortest axis is 600px, meaning the input would for example be 600xHx3. But if you are seeing the same error as above, chances are lowering the learning rate helps.
from mnc.
@oh233 hi~ i have trained on my own data once before normally, but now i train it again, without any change, it happend to this:
/data2/qinhaifang/MNC/tools/../lib/transform/bbox_transform.py:86: RuntimeWarning: overflow encountered in exp
pred_w = np.exp(dw) * widths[:, np.newaxis]
/data2/qinhaifang/MNC/tools/../lib/transform/bbox_transform.py:86: RuntimeWarning: overflow encountered in multiply
pred_w = np.exp(dw) * widths[:, np.newaxis]
/data2/qinhaifang/MNC/tools/../lib/transform/bbox_transform.py:87: RuntimeWarning: overflow encountered in exp
pred_h = np.exp(dh) * heights[:, np.newaxis]
/data2/qinhaifang/MNC/tools/../lib/transform/bbox_transform.py:87: RuntimeWarning: overflow encountered in multiply
pred_h = np.exp(dh) * heights[:, np.newaxis]
/data2/qinhaifang/MNC/tools/../lib/pylayer/stage_bridge_layer.py:107: RuntimeWarning: overflow encountered in exp
bottom[0].diff[i, 3] = dfdw[ind] * (delta_x +np.exp(delta_w))
/data2/qinhaifang/MNC/tools/../lib/pylayer/stage_bridge_layer.py:108: RuntimeWarning: overflow encountered in exp
then the loss will become NAN
is there anybody happend to this error?
thank you for your help in advance!
from mnc.
@hgaiser thank you! but i used to train the model normally, now it happends to this error without any change.
from mnc.
Anyone got such error? Thanks.
/mnt/sda/MNC/tools/../lib/transform/bbox_transform.py:129: RuntimeWarning: invalid value encountered in greater_equal
keep = np.where((ws >= min_size) & (hs >= min_size))[0]
Traceback (most recent call last):
File "./tools/train_net.py", line 96, in
_solver.train_model(args.max_iters)
File "/mnt/sda/MNC/tools/../lib/caffeWrapper/SolverWrapper.py", line 127, in train_model
self.solver.step(1)
File "/mnt/sda/MNC/tools/../lib/pylayer/proposal_layer.py", line 186, in backward
unmap_val = self._ind_after_filter[self._ind_after_sort[proposal_index[top_non_zero_ind]]]
IndexError: arrays used as indices must be of integer (or boolean) type
from mnc.
I don't think I have seen that error before, but what you could try is to delete the cache files (in data/cache/
) and try again. Otherwise I suggest diving into the python code to figure out where and why it messes up. Start by printing the ws
, hs
and min_size
values, apparently one of those is incorrect.
from mnc.
Lowering the learning rate can solve it. Will dive into python code if issue happens again.
from mnc.
I found the same error as above with learning rate = 0.001 and 0.0001. However, it works after changing the learning rate to 0.00001. From observing the code, it seems that the regression value of bbox location diverge and cause the overflow in backprop.
from mnc.
from mnc.
Adjusting learning rate affected the fine-tuning results. By following the instruction for end-to-end training 5-stage MNC model by VOC2012 with adjusted learning rate (0.0001), I got [email protected] = 36.16 and [email protected] = 13.08.
from mnc.
from mnc.
@souryuu Hi, I am using mnc_5stage.sh to train the model with learning rate (0.00001), I am wondering how much iteration I need? Did you use lr=0.0001 with iteration=25000 or 250000?
from mnc.
@brisker did you change the number of classes?
from mnc.
great with 0.00001
from mnc.
Related Issues (20)
- image size my own data HOT 4
- MNC performance with NVidia Pascal architecture
- caffe-MNC compile error with cuda 8 and both cudnn 5.1.10 and 6.0.21 HOT 2
- Keras version of MNC HOT 3
- Is it possible to train an unmodified version on MNC with a different dataset? HOT 2
- Creating SBD dataset from polygon information
- cpu_mask_voting v.s. gpu_mask_voting HOT 1
- Onedrive Download Error
- The issue of speed
- trained
- What is the purpose of MIX_INDEX
- Caffe compilation issues (CUDNNv5)
- caffe-mnc folder is empty HOT 1
- Error in demo HOT 1
- compile lib in windows
- How can the inference process use 5 stage when the cascade structures are for training 3 stages?
- download the trained MNC model failed
- pretrained resnet101 mnc model download
- Is there interest in Dockerizing the MNC code? HOT 2
- Will the code for BoxSup be released?
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from mnc.