training error with my own datasets with RuntimeWarning, Help? about mnc HOT 17 CLOSED

Lower the learning rate.

from mnc.

@hgaiser Thanks ! Besides,
I want to train from scratch without finetuned model, when I comment the "weights-" line in mnc_5stage.sh, why the the training data becomes the voc2007? I just want to train my own dataset and put it all in the VOCdevkitSDS folder

from mnc.

Why do you want to train from scratch? My advice is to use the pretrained model actually.

from mnc.

@hgaiser hi~ when i change the input data size to 1_3_1200*1200, it happend to this error and the loss become NaN. i want to know why it happends?and low the learning rate could be work?

from mnc.

Not sure what you mean with that input size. Images are by default resized such that the shortest axis is 600px, meaning the input would for example be 600xHx3. But if you are seeing the same error as above, chances are lowering the learning rate helps.

from mnc.

@oh233 hi～ i have trained on my own data once before normally, but now i train it again, without any change, it happend to this:

/data2/qinhaifang/MNC/tools/../lib/transform/bbox_transform.py:86: RuntimeWarning: overflow encountered in exp
pred_w = np.exp(dw) * widths[:, np.newaxis]
/data2/qinhaifang/MNC/tools/../lib/transform/bbox_transform.py:86: RuntimeWarning: overflow encountered in multiply
pred_w = np.exp(dw) * widths[:, np.newaxis]
/data2/qinhaifang/MNC/tools/../lib/transform/bbox_transform.py:87: RuntimeWarning: overflow encountered in exp
pred_h = np.exp(dh) * heights[:, np.newaxis]
/data2/qinhaifang/MNC/tools/../lib/transform/bbox_transform.py:87: RuntimeWarning: overflow encountered in multiply
pred_h = np.exp(dh) * heights[:, np.newaxis]
/data2/qinhaifang/MNC/tools/../lib/pylayer/stage_bridge_layer.py:107: RuntimeWarning: overflow encountered in exp
bottom[0].diff[i, 3] = dfdw[ind] * (delta_x +np.exp(delta_w))
/data2/qinhaifang/MNC/tools/../lib/pylayer/stage_bridge_layer.py:108: RuntimeWarning: overflow encountered in exp
then the loss will become NAN
is there anybody happend to this error?
thank you for your help in advance!

from mnc.

@hgaiser thank you! but i used to train the model normally, now it happends to this error without any change.

from mnc.

Anyone got such error? Thanks.

/mnt/sda/MNC/tools/../lib/transform/bbox_transform.py:129: RuntimeWarning: invalid value encountered in greater_equal
keep = np.where((ws >= min_size) & (hs >= min_size))[0]
Traceback (most recent call last):
File "./tools/train_net.py", line 96, in
_solver.train_model(args.max_iters)
File "/mnt/sda/MNC/tools/../lib/caffeWrapper/SolverWrapper.py", line 127, in train_model
self.solver.step(1)
File "/mnt/sda/MNC/tools/../lib/pylayer/proposal_layer.py", line 186, in backward
unmap_val = self._ind_after_filter[self._ind_after_sort[proposal_index[top_non_zero_ind]]]
IndexError: arrays used as indices must be of integer (or boolean) type

from mnc.

I don't think I have seen that error before, but what you could try is to delete the cache files (in data/cache/) and try again. Otherwise I suggest diving into the python code to figure out where and why it messes up. Start by printing the ws, hs and min_size values, apparently one of those is incorrect.

from mnc.

Lowering the learning rate can solve it. Will dive into python code if issue happens again.

from mnc.

I found the same error as above with learning rate = 0.001 and 0.0001. However, it works after changing the learning rate to 0.00001. From observing the code, it seems that the regression value of bbox location diverge and cause the overflow in backprop.

from mnc.

from mnc.

Adjusting learning rate affected the fine-tuning results. By following the instruction for end-to-end training 5-stage MNC model by VOC2012 with adjusted learning rate (0.0001), I got [email protected] = 36.16 and [email protected] = 13.08.

from mnc.

from mnc.

@souryuu Hi, I am using mnc_5stage.sh to train the model with learning rate (0.00001), I am wondering how much iteration I need? Did you use lr=0.0001 with iteration=25000 or 250000?

from mnc.

@brisker did you change the number of classes?

from mnc.

great with 0.00001

from mnc.