Attempted PyTorch implementation of Active Neural Generative Coding (ANGC) from Backprop-Free Reinforcement Learning with Active Neural Generative Coding. This builds on Neural Generative Coding (NGC) from The neural coding framework for learning generative models and applies it to RL problems.

My notation differs in spots from the paper. One major difference is that I use row-major convention (1-d vectors are row vectors instead of column vectors), so that I don't have to worry about transposing things when going from math to implementation.

There's an inference phase that iterates, for timesteps $t = 1, \dots, T$:

$$\bar{z}^\ell(t) = g^{\ell}(\phi^{\ell + 1}[z^{\ell + 1}(t)] \cdot W^{\ell + 1})$$

$$e^\ell(t) = \frac{1}{2 \beta_e} (\phi^\ell[z^\ell(t)] - \bar{z}^\ell(t))$$

$$d^\ell(t) = -e^{\ell}(t) + e^{\ell-1}(t) E^{\ell}$$

$$z^\ell(t) = z^\ell(t-1) + \beta (- \gamma_v z^{\ell}(t) + d^{\ell}(t))$$

(where the last equation is slightly simplified to exclude the NGC lateral term, which is not used in ANGC):

TODO pseudocode

$$\Delta W^\ell = \phi^\ell[z^{\ell + 1}]^\top e^{\ell} \odot M_W^\ell$$

$$\Delta E^\ell = \gamma_e (\Delta W^{\ell})^\top \odot M_E^\ell$$