Add derivatives for convolution about xdiff.jl HOT 5 CLOSED

For future reference, in Einstein notation 2D convolution of matrix X with filter W may be expressed as:

Y[i,j] = W[m,n] * X[i+m-1, j+n-1]

The derivative dY[i,j] / dW[m,n] is trivially inferred to be:

dY[i,j] / dW[m,n] = X[i+m-1, j+n-1] 

And for a scalar cost C the derivative is:

dC / dW[m,n] = X[i+m-1, j+n-1]

Given that according to Einstein notation indices i and j are summed out, we essentially get a convolution of ones(size(W)) with "filter" X. If I haven't forgotten any important details, adding derivatives w.r.t. W should be trivial.

Derivative w.r.t. X may be more tricky, although I haven't thought about it yet.

from xdiff.jl.

Seems like derivative w.r.t. X is also trivial - if we seek a derivative dY[i,j] / dX[p, q], we can substitute:

• p = i + m - 1
• q = j + n - 1

and thus express m and n as:

• m = p - i + 1
• n = q - j + 1

which gives us an alternative form for convolution:

Y[i,j] = W[p-i+1, q-j+1] * X[p, q]

and derivatives:

dY[i,j] / dX[p, q] = W[p-i+1, q-j+1]
dC / dX[p, q] = W[p-i+1, q-j+1]

And if I'm not mistaken, the last one is a convolution of ones(size(X)) with filter flip(W).

from xdiff.jl.

Also, if we add stride s, convolution becomes:

Y[i,j] = W[m,n] * X[s*i+m-1, s*j+n-1]

This has been described in 1 along with the equation for pooling operations. E.g. max pooling may be described as:

Y[i,j] = max(W[m,n], X[s*i+m-1, s*j+n-1])

from xdiff.jl.

Basic differentiation of convolutions is now in conv branch.

from xdiff.jl.

Done for conv2 and VectorCodeGen, although we need more sophisticated version for serious convolution stuff (e.g. support strides as TensorFlow does it).

from xdiff.jl.