question about gradient calculation with respect to weight about sphereface HOT 4 CLOSED

also, since you normalize the weight by overwriting in forward() instead of keeping the original weight like here:

  Dtype* norm_weight = this->blobs_[0]->mutable_cpu_data();
  Dtype temp_norm = (Dtype)0.;
  for (int i = 0; i < N_; i++) {
  	temp_norm = caffe_cpu_dot(K_, norm_weight + i * K_, norm_weight + i * K_);
  	temp_norm = (Dtype)1./sqrt(temp_norm);
  	caffe_scal(K_, temp_norm, norm_weight + i * K_);
  }

so everytime the after update of parameter: weight = weight + grad_w, the weight is simply clipped into the normalized version?

from sphereface.

It is actually a normalized version of the gradient, which can help converge more stably. The direction is the same as before. What we do here is simply to rescale the gradient (the learning rate can help us decide the scale). Similar idea and intuition also appear in https://arxiv.org/pdf/1707.04822.pdf.

However, if you use the original gradient to do the backprop, you could still make it work and obtain similar results, but may not be as stable as this normalized one.

from sphereface.

same question here. why update of weight it the same for m=1, 2, 3, 4

from sphereface.

+1, backward seems not correspond with forward.

from sphereface.