loss is not stable about focal-loss-keras HOT 12 CLOSED

@braindancer Agreed! @toosyou 's implementation is wrong. The focal weights should be adjusted pixel by pixel given each pixel's prediction and ground truth labels.

In summary, to make it more stable, one should:

1. Train the model use cross entropy loss for a few epochs first;
2. Add "y_pred = K.clip(y_pred, eps, 1-eps)".

from focal-loss-keras.

Add in clipping to y_pred to be between (eps,1-eps)

from focal-loss-keras.

I think it's because the calculation of cross_entropy is not stable.
I am using

def focal_loss_fixed(y_true, y_pred):
        y_true = K.flatten(y_true)
        y_pred = K.flatten(y_pred)

        alpha_factor = keras.backend.ones_like(y_true) * alpha
        alpha_factor = tf.where(keras.backend.equal(y_true, 1), alpha_factor, 1 - alpha_factor)
        focal_weight = tf.where(keras.backend.equal(y_true, 1), 1 - y_pred, y_pred)
        focal_weight = alpha_factor * focal_weight ** gamma

        cls_loss = focal_weight * K.binary_crossentropy(y_true, y_pred)
        return  K.sum(cls_loss)

then it's working fine.

from focal-loss-keras.

same problem, nan when training.

from focal-loss-keras.

very very not stable ，it is easy to turns to Nan

from focal-loss-keras.

It looks like bad initialization is the problem. What works for me is to run 2 epochs with binary_crossentropy from keras.losses, and then compile model again, using focal loss and train until convergence.

from focal-loss-keras.

You can try alpha=0.25. This make my training much more stable.

from focal-loss-keras.

I've tried adding clipping as suggested above and using alpha=0.25, but sometimes it still fails.
Why default value for alpha is 2? it makes no sense, because it should be less than 1 to weight values properly

from focal-loss-keras.

I've set the alpha=.25 as in the paper.

from focal-loss-keras.

I am still getting nan after a few epochs even with alpha=0.25 and i even added k.epsilon() in both logs.
Any fix recommended?

from focal-loss-keras.

@sarim-zafar I tried to find what causes this nans, but didn't succeed, maybe you can find it out

from focal-loss-keras.

@toosyou I believe yours is not the focal loss implemention since binary_crossentropy is already a scalar

from focal-loss-keras.