We all know that we should use cross-entropy loss with logistic regression.

But what if we use mean square loss?

Following are pictures of the curve MeanSquareCost(weight).

These pictures show that there will be multiple local minima if we use MeanSquareCost.

We all know that we should random initialize neural network before starting to train.

But what if we initialize neural network with 0?

The demo shows that the weight of neurons in the same layer will be the same, which make them perform the same function, and this symmetry won't be broken during training.