After few step Loss explodes/vanishes and than in the later epochs it produces 'Nan' as loss output. about focal-ctc-omr HOT 17 CLOSED

Good idea! I try to clip ctc loss and not clip gradient, loss may converge now

from focal-ctc-omr.

Problem: After few step Loss explodes and than in the later epochs it produces 'Nan' as loss output.
It is suffering from exploding gradients problem!
Solution: Gradient Clipping!

from focal-ctc-omr.

Try this and tell me if the problem is reoccurring

Gradient Clipping:

Apply: tf.clip_by_value(clipping_variable,1e-10,1.0)
logits=tf.clip_by_value(logits,1e-10,1.0)

from focal-ctc-omr.

Thanks for your help, loss explodes problem is solved after add Gradient Clipping, but loss may not converge

from focal-ctc-omr.

Try different Learning rates and alpha, gamma values

from focal-ctc-omr.

I change some learning rates and alpha, gamma values and it not work. And i print p and ctc_loss ,p is always 0 and ctc_loss is always same, but when i set gamma to 0, ctc_loss becomes normal.

from focal-ctc-omr.

Try:
logits=tf.clip_by_value(logits,1e-7,1.0-1e-7)
or clipping is required near power function in the line where it calculates gamma

from focal-ctc-omr.

maybe clip as below is ok，ctc loss is not always same, but it is still not converge

from focal-ctc-omr.

What about p values? Are they still zero?
If they are still zero then try different function to calculate exp().

from focal-ctc-omr.

most of it is zeros and some of it may be 1e-37,i have try tf.math.exp() instead of tf.exp, result is same and not converge

from focal-ctc-omr.

Upper Limit of clipping would be an issue.
As the values are very small in p there might be issue in upper limit of clipping.
Try printing values of p without clipping.
Then deduce which range of values would be good for clipping.
Once you get values of p in some appropriate range (and not 0 or very small number) after applying clipping, it should converge.

from focal-ctc-omr.

In fact, loss becomes nan when p values is still zeros without clipping, so I can not get values of appropriate range

from focal-ctc-omr.

Now, I try to train only with ctc loss unitl converage and fintinue with focal loss, and i will update while i get result

from focal-ctc-omr.

Also keep a check on ctc_loss output range.

from focal-ctc-omr.

Final Solution:
Clip ctc_loss() instead of gradients.
I am closing this issue now!

from focal-ctc-omr.

@cjt222 hi,does FocalCtcLoss improve your CRNN accuracy?

from focal-ctc-omr.

@cjt222 hi,does FocalCtcLoss improve your CRNN accuracy?

I do not impore CRNN accuracy with FocalCtcLoss ,I do not know if I can not adjust a better alpha and gamma

from focal-ctc-omr.