[Question]: RL Project, null gradient with GradientTape about tensorflow.net HOT 9 CLOSED

First, you get criticc_gradient but didn't use it, I think it could be deleted here, https://github.com/dryang21/RL_IN_CSHARP/blob/9f70aeb8045d965fee09a3d76f956d3eb342149f/IPRL_debug.cs#L302C1-L303

Second, it will invoke stop_recording() after invoke tape.gradient, and then pop the _tapeSet and the _tapeSet will be empty. You call tape.gradient twice in each for loop, for a total of six times, each using the same tape. So you have been using an empty _tapeSet after calling the first tape.gradient, so it will cause an error. So you should use a new tape every time you call tape.gradient. You can refer to this example:

If there still exsit any problems, please let me know.

from tensorflow.net.

You can use model.Apply(input)

from tensorflow.net.

Thanks for the quick replay, I really appreciate it. But the null gradient issue persists even after I change to model.Apply() . I was trying to perform a line-by-line translation of https://keras.io/examples/rl/actor_critic_cartpole/ the keras example from python to c#. All works smoothly except the tape.gradient(). I would greatly appreciate it if you could take the time to review my work and check if there are any errors or mistakes. The code is attached as follows:
https://github.com/dryang21/RL_IN_CSHARP/blob/main/IPRL_debug.cs

from tensorflow.net.

Is it possible to creat a branch and commit to GitHub? That will save us time to reproduce the issue.

from tensorflow.net.

Is it possible to creat a branch and commit to GitHub? That will save us time to reproduce the issue.

Thanks for the feedback.I have created a new repository and uploaded the .cs file. https://github.com/dryang21/RL_IN_CSHARP

from tensorflow.net.

First, you get criticc_gradient but didn't use it, I think it could be deleted here, https://github.com/dryang21/RL_IN_CSHARP/blob/9f70aeb8045d965fee09a3d76f956d3eb342149f/IPRL_debug.cs#L302C1-L303

Second, it will invoke stop_recording() after invoke tape.gradient, and then pop the _tapeSet and the _tapeSet will be empty. You call tape.gradient twice in each for loop, for a total of six times, each using the same tape. So you have been using an empty _tapeSet after calling the first tape.gradient, so it will cause an error. So you should use a new tape every time you call tape.gradient. You can refer to this example:

TensorFlow.NET/test/TensorFlowNET.UnitTest/Training/BasicLinearModel.cs

Lines 42 to 61 in 9e3654b

	Func<Tensor, Tensor, float, Tensor> train = (inputs, outputs, learning_rate)
	=>
	{
	using var t = tf.GradientTape();
	var current_loss = loss(outputs, model(inputs));
	var (dW, db) = t.gradient(current_loss, (W, b));
	W.assign_sub(learning_rate * dW);
	b.assign_sub(learning_rate * db);
	return current_loss;
	};
	var epochs = range(10);
	foreach (var epoch in epochs)
	{
	var current_loss = train(inputs, outputs, 0.1f);
	print($"Epoch {epoch}: W={(float)W.numpy()} b={(float)b.numpy()}, loss={(float)current_loss.numpy()}");
	if (epoch > 0) // skip first epoch
	Assert.IsTrue((bool)(current_loss < init_loss));
	}

If there still exsit any problems, please let me know.

Thanks for the suggestion. I have updated the code with: https://github.com/dryang21/RL_IN_CSHARP/blob/9f70aeb8045d965fee09a3d76f956d3eb342149f/IPRL_debug.cs

1. initiate tape every epoch and call gradient only once for each tape
   However, when I called gradient, the exception happened "stack empty". I tried to debug the process, I found that the count of the _tapeset was always zero even right after initialization, and the value of _tape showed "t_a._tape threw an exception of the type 'System.invalidOperationException' ".

Thank you for your assistance with this matter! I sincerely appreciate your time and help

The version setting:
1)visual studio 2019, c#7.3, .NET framework 4.7.2
(vs2022 could not be installed due to clinical computer restriction)

from tensorflow.net.

Hello, the code runs successfully after changing this line of code (https://github.com/dryang21/RL_IN_CSHARP/blob/1d48daa3952aa9b686d8751a5eb35a15bf2cfe70/IPRL_debug.cs#L263) to the following:

using (Tensorflow.Gradients.GradientTape t_a = tf.GradientTape(), t_b = tf.GradientTape())

Howerver, the loss didn't reduce and the value of reward didn't increase. I'm not familiar with RL so I'm not sure if this result of this code is correct.

from tensorflow.net.

Hello, the code runs successfully after changing this line of code (https://github.com/dryang21/RL_IN_CSHARP/blob/1d48daa3952aa9b686d8751a5eb35a15bf2cfe70/IPRL_debug.cs#L263) to the following:

using (Tensorflow.Gradients.GradientTape t_a = tf.GradientTape(), t_b = tf.GradientTape())

Howerver, the loss didn't reduce and the value of reward didn't increase. I'm not familiar with RL so I'm not sure if this result of this code is correct.

It works, thanks! Though it is a little weird that the gradient only works when using tensorflow.binding.

from tensorflow.net.

Close due to this problem has been solved.

from tensorflow.net.