Hey Ethan
My name is Yi Zheng. Currently I also implemented the e2c using tensorflow e2c implementation.
To test the model, I first turn off the dynamics part, so the model is just a vae, and I tested it on MINST, the results look alright:

Then I use your code to generate the simulated pendulum image from gym, and put them to the model, the results using the sam Adam optimizer, set the hyperparams according to the , training epoch are like this:

So the vae pretty much failed completely on the inverted pendulum data-set, and I double-checked my code but couldn't find any problem. And I could figure out why the model can work on a set of 0-1 images but not another. Did you test your implementation on the inverted pendulum dataset ? How are the results like ?