Metrics of algorithm performance about mushroom-rl HOT 7 CLOSED

After a long time, a lot of work, and many trials, we are finally proud to announce our first stable version of the MushroomRL Benchmark:
https://github.com/MushroomRL/mushroom-rl-benchmark/tree/1.0.0

You can use this library along with Mushroom-1.6.1 to reproduce the benchmarks.
Results, used parameters, and documentation can be found on our Readthedocs page:
https://mushroom-rl-benchmark.readthedocs.io/en/latest/?badge=latest

Thank you very much for your patience and for the help!
Thanks also to @benvoe for creating the first draft of this package.

ps: @KeAWang I hope that this solves your issues.

from mushroom-rl.

Ok, the work took much more than expected, but finally we are ready to open source MushroomRL Benchmark!
https://github.com/MushroomRL/mushroom-rl-benchmark

Many thanks to our student @benvoe for the effort and the dedication to this work.
Unfortunately, is still "work in progress", and we have to check that everything is working, tune some algorithms, fix some computational issues we have found.

Also, I'll soon public the results of the benchmarks in ReadTheDocs with the used parameters. Stay tuned for more updates and news!

from mushroom-rl.

dear @angel-ayala,
Unfortunately, due to the coronavirus pandemic, we are a bit delayed on the benchmarking task.
We are currently finishing the benchmark and running the experimental campaign to provide basic metrics on many state-of-the-art benchmarking environments.
We hope that we will have some results soon. We will publish the benchmarking suite as soon as it's ready.
This will take a bit of time, but we expect that it will be ready by the next month.

from mushroom-rl.

Thanks for your interest in MushroomRL.

We are aware that the major drawback of MushroomRL is the lack of benchmarks for experimental results. We are currently working to solve the issue and to provide more than the benchmark you are asking for. We are implementing an extensive (and extendable) benchmarking system with statistically significant plots in different environments.
Unfortunately, we need some time to do scientifically solid work, but we expect to get some results in the next few months.

We are also working to implement most of the common "tricks" for Deep RL algorithms.
Some of them are already in the dev branch (e.g. state normalization). More will come.
Our major concern is to maintain a clean and understandable codebase: the main objective of MushroomRl is to be modular and easy to use, to allow fast development of new RL algorithms and boost Deep RL research.

However, there will be always some major differences between MushroomRL and other frameworks.
We will always treat separately absorbing states and truncated trajectories (something that is not happening in raw OpenAI gym environments).
Furthermore, at least in our baseline algorithm implementations, for readability, we will never support shared network architectures (between actor and critic networks). However, if needed, these architectures can be easily implemented by the user.

Maybe in the future, we will add a tutorial on how to implement a shared network version of PPO.

We need help from the community! We will glad to fix issues reported by the users, improve the algorithms by adding "tricks" (if they don't break the theoretical properties of the algorithm and code structure) and to add other quality of life improvements.

from mushroom-rl.

Amazing! Thank you so much for this great package!

from mushroom-rl.

Hi!,
First of all, congrats by this package it seems to be very simple and modular enough to understand how works and run and RL algorithm.
I'm working in a project now, and I'm intended to use it, so I must run some test modify some aspects like policy, but most of all are the metrics to benchmark the algorithm, do you have posted anywhere the tasks to be done to accomplish that?
I would like to help you guys, but I'm requiring some guidelines to make a contribution here.

Regards,

from mushroom-rl.

Thank you!

from mushroom-rl.