abhisheksuran / reinforcement_learning Goto Github PK

View Code? Open in Web Editor NEW

94.0 94.0 36.0 363 KB

Deep Reinforcement Learning Algorithms implemented with Tensorflow 2.3

Jupyter Notebook 99.24% Python 0.76%

actor-critic atari policy-gradient ppo reinforcement-learning reinforcement-learning-algorithms tensorflow2

reinforcement_learning's People

Contributors

Stargazers

Watchers

reinforcement_learning's Issues

Not able to open Reinforcement_Learning.ipynb file

Not able to open Reinforcement_Learning.ipynb file, can you please fix this issue?

About running the code

Hello! I wonder what gpu do you use to train the dqn on? How many of them?
Have you successfully trained the network? If so, could you tell me how long does it take and how is the result?
Thank you very much!!

PPO ratio between old policy and new policy

Hi @abhisheksuran, thank you for the amazing implementation. Seems like the ratio between the old probability and new probability will always be 1 as they are always the same in the PPO code. I used the following approach to be sure they are likely to be different ( see lines commented with many #######). I may be wrong,. If so! Please enlighten me.

class agent():
    def __init__(self, gamma = 0.99):
        self.gamma = gamma
        # self.a_opt = tf.keras.optimizers.Adam(learning_rate=1e-5)
        # self.c_opt = tf.keras.optimizers.Adam(learning_rate=1e-5)
        self.a_opt = tf.keras.optimizers.Adam(learning_rate=7e-3) ### 3e-3
        self.c_opt = tf.keras.optimizers.Adam(learning_rate=7e-3) ### 3e-3
        self.actor = actor()
        self.critic = critic()

        self.old_probs = None ####################### This variable will stock old probability

        self.clip_pram = 0.2

And then in the function learn:

def learn(self, states, actions,  adv , discnt_rewards):  ############# no needs of old_probs as parameter
        discnt_rewards = tf.reshape(discnt_rewards, (len(discnt_rewards),))
        adv = tf.reshape(adv, (len(adv),))
        
        old_probs = self.old_probs ##################### assign old_probs

        old_p = old_probs ### I didn't see where you used old_p in following lines

        old_p = tf.reshape(old_p, (len(old_p),2))
        with tf.GradientTape() as tape1, tf.GradientTape() as tape2:
            p = self.actor(states, training=True)
            v =  self.critic(states,training=True)
            v = tf.reshape(v, (len(v),))
            td = tf.math.subtract(discnt_rewards, v)
            c_loss = 0.5 * kls.mean_squared_error(discnt_rewards, v)
            a_loss = self.actor_loss(p, actions, adv, old_probs, c_loss)
         
        self.old_probs = p.numpy() ################## Keep previous probability before applying the gradient to the actor

        grads1 = tape1.gradient(a_loss, self.actor.trainable_variables)
        grads2 = tape2.gradient(c_loss, self.critic.trainable_variables)
        self.a_opt.apply_gradients(zip(grads1, self.actor.trainable_variables))
        self.c_opt.apply_gradients(zip(grads2, self.critic.trainable_variables))
        return a_loss, c_loss

Then to be sure to have a set of of old_probability for the first run

  value = agentoo7.critic(np.array([state])).numpy()
  values.append(value[0][0])
  np.reshape(probs, (len(probs),2))
  probs = np.stack(probs, axis=0)

  if s == 0:   ########################## set old_probabilities  to the same as actual probabilities at the first run
    agentoo7.old_probs = np.copy(p)

  states, actions,returns, adv  = preprocess1(states, actions, rewards, dones, values, 1)
  for epocs in range(10):
     al,cl = agentoo7.learn(states, actions, adv, returns) ########### no need of probs as parameter

Wrong formatting of Reinforce_(PG).ipynb

Hi, your code for "Reinforce_(PG).ipynb" seems to have some problems with the formatting so it does not show properly:

Build keras model

In Specifying the input shape in advance | TensorFlow Core, it says:

Generally, all layers in Keras need to know the shape of their inputs in order to be able to create their weights.

It then gives two ways to specify the shape of inputs:

Start your model by passing an Input object to your model, so that it knows its input shape from the start:

model = keras.Sequential()
model.add(keras.Input(shape=(4,)))
model.add(layers.Dense(2, activation="relu"))

A simple alternative is to just pass an input_shape argument to your first layer:

model = keras.Sequential()
model.add(layers.Dense(2, activation="relu", input_shape=(4,)))

But I haven't seen input shapes defined in your code:

(Reinforce)Policy Gradient with TensorFlow2.x

class model(tf.keras.Model):
  def __init__(self):
    super().__init__()
    self.d1 = tf.keras.layers.Dense(30,activation='relu')
    self.d2 = tf.keras.layers.Dense(30,activation='relu')
    self.out = tf.keras.layers.Dense(env.action_space.n,activation='softmax')

  def call(self, input_data):
    x = tf.convert_to_tensor(input_data)
    x = self.d1(x)
    x = self.d2(x)
    x = self.out(x)
    return x

Proximal Policy Optimization (PPO) With TensorFlow 2.x

class critic(tf.keras.Model):
  def __init__(self):
    super().__init__()
    self.d1 = tf.keras.layers.Dense(128,activation='relu')
    self.v = tf.keras.layers.Dense(1, activation = None)

  def call(self, input_data):
    x = self.d1(input_data)
    v = self.v(x)
    return v
    

class actor(tf.keras.Model):
  def __init__(self):
    super().__init__()
    self.d1 = tf.keras.layers.Dense(128,activation='relu')
    self.a = tf.keras.layers.Dense(2,activation='softmax')

  def call(self, input_data):
    x = self.d1(input_data)
    a = self.a(x)
    return a

You just call self.d1(input_data) directly, does tensorflow detect the input shape implicitly?

Besides, in your PG code you did tf.convert_to_tensor(input_data) before calling self.d1(x), while you didn't do that in PPO code. Does this mean tf.convert_to_tensor() is unnecessary?

Reinforce_(PG).ipynb not valid

hi, Reinforce_(PG).ipynb is not valid, can you reupload the file? thanks

abhisheksuran / reinforcement_learning Goto Github PK

reinforcement_learning's People

Contributors

Stargazers

Watchers

Forkers

reinforcement_learning's Issues

Not able to open Reinforcement_Learning.ipynb file

About running the code

PPO ratio between old policy and new policy

Wrong formatting of Reinforce_(PG).ipynb

Build keras model

Reinforce_(PG).ipynb not valid

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent