Comments (9)
I think in this implementation they use softmax as the output activation function when sampling action.
And see the code below, you can find that they have attempted to use Argmax activation by return CategoricalPdType(ac_space.n)
when sampling. But eventually they use softmax activation when training the Q net.
def make_pdtype(ac_space):
from gym import spaces
if isinstance(ac_space, spaces.Box):
assert len(ac_space.shape) == 1
return DiagGaussianPdType(ac_space.shape[0])
elif isinstance(ac_space, spaces.Discrete):
# return CategoricalPdType(ac_space.n)
return SoftCategoricalPdType(ac_space.n)
elif isinstance(ac_space, spaces.MultiDiscrete):
#return MultiCategoricalPdType(ac_space.low, ac_space.high)
return SoftMultiCategoricalPdType(ac_space.low, ac_space.high)
elif isinstance(ac_space, spaces.MultiBinary):
return BernoulliPdType(ac_space.n)
else:
raise NotImplementedError
from maddpg.
Hello, @pengzhenghao!
I've looked into involved functions again, I guess they use SoftCategoricalPdType(ac_space.n), then SoftCategoricalPdType.sample() to somehow add noise to actions, finally to softmax(logits - noise) as output of actor network.
And the noise added to action is from:
def sample(self):
u = tf.random_uniform(tf.shape(self.logits))
return U.softmax(self.logits - tf.log(-tf.log(u)), axis=-1)
I don't quite get it why they handle the noise in this way.
from maddpg.
The sample function in distribution is implementation of Gumbel-softmax, I added it to my code, now it helps to speed up stabilize the training, but my speaker still can not tell the different landmarks.
How do you handle the action exploration then?
from maddpg.
Hello, @pengzhenghao!
I don't quite get it why they handle the noise in the form of the log-log link function. Due to the log-log link function is non-linear function, the noise randomly generated every time could fluctuate, how to control the degree of noise to ensure adequate action exploration?
from maddpg.
Hello, @pengzhenghao!
I don't quite get it why they handle the noise in the form of the log-log link function. Due to the log-log link function is non-linear function, the noise randomly generated every time could fluctuate, how to control the degree of noise to ensure adequate action exploration?
Gumble-Softmax Trick is an important re-parameterization trick that can help smoothing the back propagation. I refer you to search with keyword "gumbel softmax" for more information. I am sorry for not providing more info since I do not thoroughly understand the whole process of gumbel softmax...
from maddpg.
Hello @djbitbyte!
You said that gumbel softmax helps to speed up stabilize the training. I am trying to reproduce the results in pytorch and using torch.nn.functional._gumbel_softmax_sample while sampling the action for current state as follows:
Also I am using torch.nn.functional.gumbel_softmax for computing target actions for next states and for computing action for current agent to be fed into actor_local.
Based on the original code and the algorithm, I am not able to understand why training is not converging after i use gumbel_softmax.
Thanks in advance!
from maddpg.
Hello!
Just to mention that there were many other issues in the code instead of gumbel-softmax because of which the training was not converging.
from maddpg.
Hello!
Just to mention that there were many other issues in the code instead of gumbel-softmax because of which the training was not converging.
Hi, I'm trying to understand how to use gumbel_softmax
in pytorch to reproduce the results. I'm using PPO but it cannot even learn the task for only one agent and one landmark completely. It reaches some good level but it's not as good as MADDPG at all. I think the problem is with this simple softmax and Categorical distribution I use and want to change it to humble softmax. I used:
policy_dist = distributions.Categorical(F.gumbel_softmax(policy_logits_out, tau=1, hard=False).to("cpu"))
But didn't get good results. There is also a distribution.Gumble
in pytorch. I think I use them incorrectly.
Can you provide an example to use them in your own algorithm?
Thank you
from maddpg.
The sample function in distribution is implementation of Gumbel-softmax, I added it to my code, now it helps to speed up stabilize the training, but my speaker still can not tell the different landmarks.
How do you handle the action exploration then?
can you provide the code of your implementation of Gumbel-softmax? I meet the same problem when using MADDPG.
many thanks.
from maddpg.
Related Issues (20)
- How to turn continuous action into discrete action HOT 1
- How to normalize the data in table of Appendix to obtain Figure 3 in paper? HOT 1
- how to evaluate maddpg?
- how to evaluate maddpg?
- Using any scenario rather than the "simple" one gives error during loading the model after training
- ../../..
- Is policy ensambles implemented in this repository?
- All scenario will fail but except scenario simple HOT 1
- How to run ddpg
- Question about the way how to update actor HOT 1
- Why ‘u_action_space = spaces.Discrete(world.dim_p * 2 + 1)’? HOT 2
- Training EVERY step, not every 100 HOT 1
- A question about maddpg.py
- A issue about the saving the model in train.py
- Loading previous state... HOT 2
- DDPG action space vs multi-agent particle environment action space
- ModuleNotFoundError: No module named 'maddpg.common' HOT 1
- How to save the videos of each 1000th episode ?
- python3.5.4已经进行维护。
- MADDPG for custom environment
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from maddpg.