Hello. Are you quite sure that history of optimizer (e.g. moments) should be zeroed at the beginning of each epoch?

Derivative of ReLU equal to 0 if x<0 and equal 1 if x>0, isn't it?

Hi, thanks for your work. I learned a lot from your blog and code!

In your gradient test code, I found that there may be something wrong in the back propagation
for dX of CNN, test code is as blow and data used in the code are here:npy.zip

w = np.load('w.npy')
b = np.load('b.npy')
dout = np.load('dout.npy')
x = np.load('x.npy')

c_layer = Conv((1, 28, 28),n_filter=32,h_filter=3,w_filter=3,stride=1,padding=1)
c_layer.W = w
c_layer.b = b
dx_num = numerical_gradient_array(lambda x: c_layer.forward(x), x, dout)
dw_num = numerical_gradient_array(lambda w: c_layer.forward(x), w, dout)
db_num = numerical_gradient_array(lambda b: c_layer.forward(x), b, dout)

out = c_layer.forward(x)
dx,grads = c_layer.backward(dout)
dw,db = grads
print("Testing backward pass of Conv Layer")
print("dX error: ",rel_error(dx,dx_num))
print("dW error: ",rel_error(dw,dw_num))
print("db error: ",rel_error(db,db_num))

the results is as blow:

Testing backward pass of Conv Layer
dX error: 1.0
dW error: 4.938012368517188e-11
db error: 2.0764855776951717e-07

I put the dataset manually in ./data/mnist.pkl.gz and run the python run_cnn.py mnist, but it gives an error:

Traceback (most recent call last):
File "run_cnn.py", line 46, in
learning_rate=0.01, X_test=X_test, y_test=y_test)
File "/Users/JianGuo/PycharmProjects/deepnet/deepnet/solver.py", line 83, in sgd_momentum
minibatches = get_minibatches(X_train, y_train, minibatch_size)
File "/Users/JianGuo/PycharmProjects/deepnet/deepnet/solver.py", line 12, in get_minibatches
X, y = shuffle(X, y)
TypeError: 'bool' object is not callable

I doubt that this would be a compatible issue with the sklearn, but even I had installed the old version of scikit-learn(0.15), it turns out the same result.

Python version:

Python 3.6.3 (default, Oct 4 2017, 06:09:15)
[GCC 4.2.1 Compatible Apple LLVM 9.0.0 (clang-900.0.37)] on darwin
Type "help", "copyright", "credits" or "license" for more information.

pip version:

pip 9.0.1 from /Users/JianGuo/PycharmProjects/deepnet/.env/lib/python3.6/site-packages (python 3.6)

requirements.txt:

numpy==1.11.3
scipy==0.16.1
matplotlib==1.5.0
ipykernel==4.2.2
ipython==4.0.1
ipython-genutils==0.1.0
ipywidgets==4.1.1
scikit-learn==0.15

Did I miss something here?

Hi,
I first used a simple three-tier CNN structure without dropout. The accuracy rate was over 90%. However, after I used dropout, the accuracy rate dropped to 40%. Is there any problem with my data set or is it another problem?

thank you
I

Hi, thanks for your share.
I found some wrong in the implementation of BatchNorm。you compute mean and var for every batch, rather than for every channels

cifar10's test accuracy does not go above 33%. What can be done?

Python2 or Python3?