sergeivolodin / causality-disentanglement-rl Goto Github PK

Problem: https://docs.google.com/presentation/d/1iYfbMrMZUn88cet36qe0h-2oHG9HpbFfEZHTVp4xlik/edit

Possible solution:

1. train the D with sparsity loss on the model + reconstruction loss where the model is fixed
2. train the M with sparsity loss + reconstruction loss where D is fixed

• Different losses: reconstruction or matrix norm or both
• Feature space or observation space fit loss and reconstruction loss
• Two, three or one optimizer [tried manually]
• Learning rate, dimension search
• Annealer grid search
• Batch norm / no batch norm after decoder
• Add a metric computing the losses with thresholded model
• Norm order p
• Projection or loss term

• Currently, lamp switches if the player is at its position
• In the original environment, there was a separate button for that.

Now in 761e211 DQN fails to converge on an environment with a 2x2 linear transform, while it's fine without the transform. The reason might be either that there were too few steps given (now 100 steps x 256 episodes, or that there is a problem with the environment being non-Markov).

Concrete step to try: try a linear q-network and see if it converges to different results with and without transform.

Next step: try policy gradients as they do not require to learn the value (which might not exist if the environment is not Markov), only the policy (which is simple -- compare two numbers and choose a corresponding action)

@jbrea adding you to the issue so that you learn faster about the progress.