with something like:

X = rand(...)
Y = rand(...)
Ze = zeros(...)
Zb = zeros(...)
eval(ein_expr)
eval(buf_expr)
@assert Ze == Zb

sizes of variables should be determined automatically from indices in Einstein notation.

Currently we generate derivative functions that are bound to input sizes. If input size changes, user needs to repeat differentiation process which may be costly. Rather than put it on the user, we can maintain a cache of functions, one per a set of input sizes, and provide the user with a "meta" function that adaptively selects needed derivative function.

We also need to keep track of buffer sizes in memory dict. The easiest way should be to replace @get_or_create with something like @get_or_alloc that also checks the size of the buffer.

This issue indirectly depends on GPU codegen.

• temporarily remove automatic parsing of nested functions - it's not stable enough, so better an exception than an implicit error
• remove "assumed" broadcasting: if there's no rule in TDIFF_RULES, arrays shouldn't be automatically broadcasted

As described in https://discourse.julialang.org/t/ann-plans-for-removing-packages-that-do-not-yet-support-1-0-from-the-general-registry/ we are planning on removing packages that do not support 1.0 from the General registry. This package has been detected to not support 1.0 and is thus slated to be removed. The removal of packages from the registry will happen approximately a month after this issue is open.

To transition to the new Pkg system using Project.toml, see https://github.com/JuliaRegistries/Registrator.jl#transitioning-from-require-to-projecttoml.
To then tag a new version of the package, see https://github.com/JuliaRegistries/Registrator.jl#via-the-github-app.

If you believe this package has erroneously been detected as not supporting 1.0 or have any other questions, don't hesitate to discuss it here or in the thread linked at the top of this post.

It would be much more pleasant to write:

struct MyModel
    W1::Matrix
    b1::Vector
    W2::Matrix
    b2::Vector
    W3::Matrix
    b3::Vector
end

predict_cost(m, x, y)
predict_cost_grad(m, x, y)

rather than

predict_cost(W1, b1, W2, b2, W3, b3, x, y)
predict_cost_grad(W1, b1, W2, b2, W3, b3, x, y)

I believe we can automatically generate intermediate functions like:

predict_cost(m_W1, m_b1, m_W2, m_b2, m_W3, m_b3, x, y)

Find derivatives for them and then generate corresponding derivative functions for both - flat arguments and structs.

Example from autoencoder derivatives:

julia> dexs[:We1]                                                                                                                                                                                                 
 quote  # /home/slipslop/.julia/v0.6/Espresso/src/from_einstein.jl, line 101:                                                                                                                                      
     ##707 = squeeze(sum(Wd, 1), 1)                                                                                                                                                                                
     ##708 = ##707 .* We2                                                                                                                                                                                          
     ##709 = squeeze(sum(##708, 1), 1)                                                                                                                                                                             
     ##710 = ##709 .* x                                                                                                                                                                                            
     dcost_dWe1 = repmat((squeeze(sum(##710, 2), 2))', size(We1, 1))                                                                                                                                               
 end                                                                                                                                                                                                               
                                                                                                                                                                                                                   
!julia> dexs[:x]                                                                                                                                                                                                   
 quote  # /home/slipslop/.julia/v0.6/Espresso/src/from_einstein.jl, line 101:                                                                                                                                      
     ##711 = -1.0                                                                                                                                                                                                  
     ##712 = squeeze(sum(Wd, 1), 1)                                                                                                                                                                                
     ##713 = ##712 .* We2                                                                                                                                                                                          
     ##714 = squeeze(sum(##713, 1), 1)                                                                                                                                                                             
     ##715 = ##714 .* We1                                                                                                                                                                                          
     ##716 = ##711 .+ ##715                                                                                                                                                                                        
     dcost_dx = transpose(squeeze(sum(##716, 1), 1))                                                                                                                                                               
 end

Here variables ##707 to ##709 compute the same thing as variables ##712 to ##714. It should be easy to rewrite all reverences of the later to references to the former and remove unused vars.

I am playing around with Espresso and XDiff, because I think it would be cool for some future version of LossFunctions.jl to generate all the losses, their derivatives, and their properties just based on defining the function (like @makeloss L(x) = 1/2 * x^2)

Taking the example of a simple popular version of the least squares loss L(x) = 1/2 * x^2, I found that rdiff changes the operation * to the broadcasted version .* which seems to prevent constant folding

julia> using Espresso, XDiff

julia> rdiff(:(1/2 * x^2), x=1.)
Dict{Symbol,Expr} with 1 entry:
  :x => :(0.5 .* (2x))

julia> simplify(:(0.5 .* (2x)))
:(0.5 .* (2x))

julia> simplify(:(0.5 * (2x)))
:x

Is this expected behaviour? How would you suggest to deal with such cases?

We need methods like foo_deriv_123(W, x, b, y) and foo_deriv_123(W, x, b, y, mem) to be able to call them using Base.invokelatest. Also may be a good starting point for #19.

Can be achieved using something like:

func_expr1 = ...
func_expr2 = ...
f = eval(mod, func_expr1)
eval(mod, func_expr2)

Currently we support:

1. Matrix multiplication
2. Summation
3. Element-wise operations and broadcasting
4. Simple convolutions

These operations work well in Einstein notation and allow to find things like Jacobians automatically. However, it's also possible to add gradient-only functions (e.g. segmented_sum) which I will call "special functions".

The tag name "v0.1" is not of the appropriate SemVer form (vX.Y.Z).
cc: @dfdx

Although preferred way to express iteration is through aggregation functions, we may still encounter loops in 3rd party code and it would be nice to support at least simple ones. One particular example is:

y = 0.0
for x in xs
    y += f(x)
end

We have 2 options here:

1. Preprocess loops transforming them into aggregation functions. E.g. loop above may be transformed into sum(f.(x)).
2. Translate a derivative of a loop into a loop of derivatives. This is more general-purpose, but looks also much harder to implement.

Required for RNN stuff.

I feel like I'm constantly doing double work to translate expressions from Einstein notation to vectorized and to buffered notations. It's not only annoying, but also error-prone. The main question is: can we remove from_einstein and use to_buffered everywhere?

So far the only place where from_einstein is actively used is in evaluate!.

Just to track the most important parts that need to be documented:

• derivative of an expression
• derivative of a function
• gradient vs. Jacobian, vectorized vs. Einstein notation
• special functions
• diff rules (@scalardiff, @tensordiff, @specialdiff)
• code generation
• GPU
• benchmarks

1. Parse broadcasting expressions (e.g. log.(x)) into a new ExNode{:broadcast}.
2. Remove support for automatic broadcasting for elementwise functions.
3. Adjust differentiation rules.

E.g.:

predict(W, b, x) = W * x + b
predict_cost(W, b, x, y) = sumabs2(predict(W, x, b) - y)

Most of this functionality is already in place, IIRC the only issue is the conflicting names of intermediate variables in nested and main functions which we may simply rename.

Just fixed a bug in eliminate_common - one of many graph transformation subroutines that are deadly hard to debug. To discover issues with such utils we can add self-consistency checks. The simplest one is to evaluate graph before and after transformation and test if some invariants hold. The price for it is a prolonged derivative compilation and so should only be done if a debug option/global var is set.

I find more and more use cases for a meta field in ExNode. One of them - to store keyword parameters to function call. For example, we could parse this:

Y = conv(X, W; stride=2)

to an ExNode like this:

ExNode
  var = Y
  ex = conv(X, W)
  meta = {kw = (stride => 2)}