Sry guys but your paper is not worth more than zero :) about rezero HOT 1 CLOSED

Dear AmorfEvo,

thank you for your constructive comments and references! We're currently revising our manuscript, including reviews and comparisons of prior work that use more complex gating mechanisms that we unfortunately missed to reference in our preprint submission.

It appears that Gated ResNet architecture contains the gate

y = (1 - a) * x + a * F(x),

while Highway Networks are

T = W_T * x + b_T 
y = T * F(x) + (1 - T) * x ,

with b_T initialized to be negative. You mention that the carry gate = 1 is an option in Highway Networks, but I see no canonical implementation we could compare to.

We will be sure to directly compare both of these architectures to ReZero and give appropriate credit.

from rezero.