Question: Global write tokens or recurrent about recurrent-memory-transformer-pytorch HOT 7 CLOSED

My thought is, if using the recurrent structure, the skip connection in transformer can implicitly make the next memory close to the previous memory.

from recurrent-memory-transformer-pytorch.

LGTM. There is one small detail that I don't know whether you need stop gradient. In Fig. 2 of the RMT paper, the gradient arrow is pointing to the write, but in Fig. 4 of the RMDT paper, the gradient arrow is point to the read. I think that won't be a big issue as long as the training is stable.

from recurrent-memory-transformer-pytorch.

@IcarusWizard ah that's interesting

would be nice if they ablated that

my take is, should make little difference, as in the first layer, the write memories have access to all the read memories

from recurrent-memory-transformer-pytorch.

@IcarusWizard but i can get that change in, to be faithful to the paper

from recurrent-memory-transformer-pytorch.

@IcarusWizard yup

do you want to see if the latest commit lines up with the paper better?

from recurrent-memory-transformer-pytorch.

@IcarusWizard i'm pretty sure it won't do anything. the way i had it before is akin to an attention pooling step on the first layer, and we know that works well from some other papers

from recurrent-memory-transformer-pytorch.

@IcarusWizard ok, i've made it a hyperparameter

thanks for reporting this!

from recurrent-memory-transformer-pytorch.