Comments (8)
Yes, it will be a drop-in replacement for Flux's current AD, and may eventually become the default.
from zygote.jl.
So the idea with Zygote is to play around with source-to-source AD techniques which are currently not in Capstan's focus. Jarrett is focused on a fairly different (and currently incompatible) implementation strategy, mainly because he has concerns about Zygote's ability to scale to handle larger programs. Right now, Zygote and Capstan will have to exist as alternatives, but Capstan is intended to become more pluggable so that it'll become possible to integrate whatever parts of Zygote work well in future.
@jrevels might wish to say if that's a fair summary, or elaborate on his concerns.
from zygote.jl.
On flux master you can do params(m)
and pass this to gradient
following Zygote's API. e.g.
m = Chain(...)
gradient(params(m)) do
m(input)
end
I haven't documented this yet because it doesn't have great coverage of Flux yet. But feel free to try it and open issues for anything you run into, and that'll help me prioritise fixes.
from zygote.jl.
Got it, thanks.
I assume you've also seen Google's Tangent? It also does source to source AD.
from zygote.jl.
Yup, Tangent is a good mental model for how Zygote is working as well. The main difficulty with doing this in Python is that you have to pretend you're working with a sufficiently-static subset of Python where you can resolve what everything actually does – otherwise you'd have to look up gradient definitions at runtime. Hence Tangent's various limitations with respect to closures, classes etc.
Zygote is fully dynamic, which works because we can actually look up gradients dynamically (via dispatch) and have Julia resolve that at compile time where possible (as usual).
Another difference is that Zygote works on SSA form IR rather than Julia source code. We don't aim for readable output in any way, but are able to support a lot of the language in a uniform way (e.g. arbitrary control flow including @goto
).
from zygote.jl.
@MikeInnes is Zygote going to be an alternative AD backend for Flux just like what you expect for capstan and cassette?
from zygote.jl.
How far are we along this path? I would like to use Zygote instead of Tracker in Flux but so far I haven't found a way of collecting parameters from more complex, i.e., chained models. But maybe I'm missing something obvious..
from zygote.jl.
Given it's been almost 2 years since Zygote became the default AD in Flux, this should be safe to close :)
from zygote.jl.
Related Issues (20)
- `gradient` broken for `(*)(::Diagonal{Real}, ::Matrix{Complex}, ::Diagonal{Real})` when updating Julia 1.8 -> 1.9 HOT 6
- Method ambiguities reported by Aqua
- slow/high allocation gradient with mapreduce and iterators HOT 11
- error in summation of product iterator HOT 2
- `sort(x; rev=true)` is not supported HOT 1
- Incorrect gradients for `plan_rfft(x) * x` HOT 2
- Gradient of scalar function of gradient giving mutating array error HOT 4
- `sum` with CUDA and view on array errors HOT 3
- Cannot take gradient of sort on 2D CuArray HOT 1
- `Ref` and broadcasting issue HOT 2
- Strip zygote frames from mutation error stack trace HOT 1
- Zygote in Julia 1.10+ not reading rrules for default constructors HOT 7
- Few issues in the Zygote Home Page documentation HOT 1
- OOM when computing the gradient in an embarrassingly parallel problem HOT 1
- Pullback over jacobian HOT 6
- `withjacobian` flattens the output when it is a matrix HOT 1
- Gradient wrt to a sparse matrix is mathematically wrong HOT 6
- Increasing memory usage in each call of gradient HOT 1
- Precompilation error in Julia nightly HOT 1
- Gradient involving `LinearAlgebra.tr` errors HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from zygote.jl.