The methods getstate() and setstate() are missing in the reps optimizer class.
When trying to pickle and controller object initialized with a reps optimizer, it throws an TypeError due to the logger object.
Adding (as in the other optimizers):

    def __getstate__(self):
        d = dict(self.__dict__)
        del d["logger"]
        return d

    def __setstate__(self, d):
        self.__dict__.update(d)
self.logger = get_logger(self, self.log_to_file, self.log_to_stdout)

should solve the problem.

https://travis-ci.org/rock-learning/bolero @malter

With a preinstalled ROS it is not possible to successfully bootstrap BOLeRo. Building urdfdom fails, probably due to some collisions with ROS headers.

Further investigation of possible options to explicitly set the used dependencies/headers/libraries appear to be necessary.

Same applies to ApproxIK which also depends on urdfdom and which also fails to build when there is already a ROS installation on the system.

This happens even if ROS isn't activated (setup.bash not sourced).

While MARSEnvironmentHelper::init() tries to hand calc_ms to mars, if(marsPlugin->control->cfg) results in segementation fault. (master)

it works with 1.4.9

This is usually not a problem because the regularization coefficient is small. It is not straightforward to fix this because the upper-level policy that does the regression does not know which context feature is always 1 and hence represents the bias.

The docstring says it can be a string but it does not say which values are valid. One of them is 'quadratic'.

Policy Search for BOLeRo

Possible Candidates

• 2-3 students from "Lernverfahren für autonome Roboter" (each student working on a different paper)

Literature

See #9

Software

• BOLeRo, tested for Ubuntu 16.04, could be used with Docker

What You Will Learn / Prerequisites

• you have to bring your own computer, we can give you a laptop for the final presentation (no PowerPoint license available!)
• policy search
• Python ecosystem: numpy, scipy, matplotlib, sklearn, pybullet

Steps

• make yourself familiar with BOLeRo
• pick one of the papers from this ticket
• implement the main algorithm and preferably example(s) / benchmark(s) from the paper
• develop on a feature branch of BOLeRo and make a pull request
• the pull request will be reviewed by us
• make a short presentation at the end of the independent studies

Evaluation Criteria

• code will be reviewed and will contribute 70% to the grade
• the final short presentation will contribute 30% to the grade
• you can get 2-6 CP for independent studies

Loading an previously saved DMP model and executing the represented trajectory fails for some trajectories.
See test case with example trajectory in commit e18402b8e4d9b9d71aae7a7bf8e77c6d41b639b5 in forked repository.
While an synthetic sinus trajectory is correctly loaded, loading and executing non-synthetic trajectories produces bad results.

There must be an issue at C++ side as all the variables on python side are correctly loaded and identical.
The acceleration ydd quickly becomes quite large at the beginning of the trajectory execution.

https://papers.nips.cc/paper/5672-model-based-relative-entropy-stochastic-search.pdf, page 2

However, the episodic REPS algorithm uses a sample based
approximation of the KL-bound, which needs a lot of samples in order to be accurate. Moreover, a
typical problem of REPS is that the entropy of the search distribution decreases too quickly, resulting
in premature convergence.

The default initial mean is 0 and the default variance is 1. If a user defines the boundaries to be [0.1, 1], the search initial distribution will not be within the boundaries. We should inform the user in this case.

You should upload a pregenerated documentation webpage on github (gh-pages) and put a link into the README file.

DMP.imitate throws error, if dt has many decimal places, e.g. 1.0/60 (60 Hertz).

• Bootstrap / pybob should check if CYTHON/cython3 is available
• Alias for pybob should be called with PYTHON/python3

Policy Search

• PI2, is already implemented #28
• PoWER, is already implemented
• NES, is already implemented #27
• ACM-ES, is already implemented #17
• Reinforce, for historical reasons
• NAC, for historical reasons
• RWR, for historical reasons
• CRKR
• PI2-CMA, might be difficult to implement
• CEM
• Evolution Strategies, recent paper
• MORE, someone is working on this
• C-MORE
• C-CMA-ES, #25
• MOTO
• NP-REPS
• Dual REPS ?
• LaDiPS
• L-BayesOpt
• PILCO
• CORL
• Black-DROPS
• Random Search

Policy Representation

• ProMP, #53
• Sample-efficient ProMP

Deep RL

• GPS
• TRPO
• GAE
• ACER
• PPO, simpler than ACER
• DDPG
• NAF
• A3C
• DQN

Interesting Papers

• A Unifying Perspective of Parametric Policy Search Methods for Markov Decision Processes

at the moment, all dependencies are installed, all source code dependencies and the current master of bolero are built, the branch is built again after that and finally the tests will be run

possible improvements:

• accelerate how dependencies are installed (cache?)
• don't build bolero twice (master + branch)

• rewrite env.sh: $AUTOPROJ_CURRENT_ROOT should only be set if it is not yet defined
• update installation instructions

We have to set the meta class of Optimizer to ABCMeta like this is done for the ContextualOptimizer.

There is an old branch mars_wo_qt. Is it still needed?

In the parameter description of the class DMPSequence, it is specified:

    initial_weights : list
        List of initial weight vectors for the DMPs

However, if initial_weights is set as a normal list, w.reshape would be called on an integer in line 109, which throws an AttributeError:

self.weights = [w.reshape(self.n_weights_per_dmp[i],
                                     self.n_task_dims)
                           for w in self.initial_weights]

When building bolero the the file _ranking_svm.c is missing which has been removed in commit 99409c5

(venv) ~/git/bolero-dev/learning/bolero/build (master)$ make install -j
Scanning dependencies of target python_setup
non-existing path in 'bolero/utils': '_ranking_svm.c'
Warning: passing language='c++' to cythonize() is deprecated. Instead, put "# distutils: language=c++" in your .pyx or .pxd file(s)
running build_ext
running build_src
build_src
building extension "bolero.utils._ranking_svm" sources
building extension "bolero.wrapper._wrapper" sources
build_src: building npy-pkg config files
customize UnixCCompiler
customize UnixCCompiler using build_ext
customize UnixCCompiler
customize UnixCCompiler using build_ext
building 'bolero.utils._ranking_svm' extension
compiling C sources
C compiler: /usr/bin/gcc -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC

compile options: '-I/media/hwiedemann/100F3CC0064B81ED/Home/git/bolero-dev/venv/lib/python3.5/site-packages/numpy/core/include -I/media/hwiedemann/100F3CC0064B81ED/Home/git/bolero-dev/venv/lib/python3.5/site-packages/numpy/core/include -I/usr/include/python3.5m -I/media/hwiedemann/100F3CC0064B81ED/Home/git/bolero-dev/venv/include/python3.5m -c'
extra options: '-O3'
gcc: _ranking_svm.c
gcc: error: _ranking_svm.c: No such file or directory
gcc: fatal error: no input files
compilation terminated.
error: Command "/usr/bin/gcc -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/media/hwiedemann/100F3CC0064B81ED/Home/git/bolero-dev/venv/lib/python3.5/site-packages/numpy/core/include -I/media/hwiedemann/100F3CC0064B81ED/Home/git/bolero-dev/venv/lib/python3.5/site-packages/numpy/core/include -I/usr/include/python3.5m -I/media/hwiedemann/100F3CC0064B81ED/Home/git/bolero-dev/venv/include/python3.5m -c _ranking_svm.c -o build/temp.linux-x86_64-3.5/_ranking_svm.o -MMD -MF build/temp.linux-x86_64-3.5/_ranking_svm.o.d -O3" failed with exit status 1
CMakeFiles/python_setup.dir/build.make:57: recipe for target 'python_setup' failed
make[2]: *** [python_setup] Error 1
CMakeFiles/Makefile2:67: recipe for target 'CMakeFiles/python_setup.dir/all' failed
make[1]: *** [CMakeFiles/python_setup.dir/all] Error 2
Makefile:138: recipe for target 'all' failed
make: *** [all] Error 2

I am trying to install bolero using bootstrap_bolero.sh inside a docker container.
The base image is python:3.7-stretch.

The commands I run to install bolero are:

apt-get update
apt-get install -y sudo
pip install pyyaml cython numpy nose 
wget https://raw.githubusercontent.com/rock-learning/bolero/master/bootstrap_bolero.sh
chmod +x bootstrap_bolero.sh
# Optionally tried to remove sudo, as the docker user is root anyways
# sed -i 's/sudo //g' bootstrap_bolero.sh
./bootstrap_bolero.sh

This seems to work fine (return code is 0), but in fact the output shows errors and bolero can not be imported after sourcing env.sh.

Errors:
 - configure: learning/bolero/include
 - configure: simulation/lib_manager
 - configure: base/cmake
 - configure: tools/configmaps
 - configure: learning/bolero/src/optimizer/cmaes_optimizer
 - configure: learning/bolero/src/bl_loader
 - configure: learning/bolero/src/environment/function_approximation
 - configure: learning/bolero/src/environment/mountain_car
 - configure: learning/bolero/src/environment/cec13_test_functions
 - configure: learning/bolero/src/optimizer/pso_optimizer
 - configure: learning/bolero/src/representation/promp
 - configure: learning/bolero/bolero/wrapper
 - configure: learning/bolero/src/representation/dmp
 - configure: learning/bolero/src/controller
 - configure: learning/bolero
Installed packages: 
[]

When trying to install by downloading the latest release from GitHub (https://github.com/rock-learning/bolero/releases/download/v1.0.0/bolero_release.zip), the install.sh runs some scripts which apparently are only Python 2 compatible, as they have print statements rather than using the print function, so that does not work well either.

Do you have any suggestions?

The full dockerfile is:

FROM python:3.7-stretch

RUN apt-get update \
    && apt-get install -y sudo \
    && pip install pyyaml cython numpy nose \
    && wget https://raw.githubusercontent.com/rock-learning/bolero/master/bootstrap_bolero.sh \
    && chmod +x bootstrap_bolero.sh \
#    && sed -i 's/sudo //g' bootstrap_bolero.sh \
    && ./bootstrap_bolero.sh

COPY entrypoint.sh /entrypoint.sh
WORKDIR ./bolero-dev
ENTRYPOINT ["/entrypoint.sh"]
CMD nosetests bolero -sv

with entrypoint.sh being:

#!/bin/bash
set -e

source ./env.sh
exec "$@"

https://mloss.org/software/view/698/

Cheat Sheet

Main issues:

• pybob only works with Python 2 rock-simulation/pybob#21
• PYTHONPATH is not set correctly by env.sh generated by pybob
• pybob only installs Python 2 packages because it does not check the Python version
  • sudo pip install scikit-learn

On Ubuntu you have to set these environment variables:

export CYTHON=cython3
export PYTHON=python3

@HWiese1980 told me that mars_environment depends on

• simulation/mars/entity_generation/smurf
• simulation/mars/smurf_loader

Both are not listed in the manifest.xml. Should we add them? @malter

We've got a Heisenbug in ProMPs. :)

See for example this build: https://circleci.com/gh/rock-learning/bolero/88, from build stage:

learning/bolero/src/representation/promp build error
...
Errors:
 - build: learning/bolero/src/representation/promp

However, unit tests of ProMPBehavior seem to be OK:

test_promp_behavior.test_smoke ... ok
test_promp_behavior.test_dmp_default_promp ... ok
test_promp_behavior.test_promp_get_set_params ... ok
test_promp_behavior.test_promp_constructor_args ... ok
test_promp_behavior.test_promp_imitate ... ok

I tried to reproduce it:

• pulled the docker image af01/bolero_dependencies:latest, cloned the repository, ran the build code; no error
• restarted the CircleCI build with SSH connection, removed the bolero-dev folder, ran bootstrap script again; no error

The error only seems to occur consistently during the first bootstrap in CircleCI. Even checked if that is an issue of pybob, but pybob only seems to report errors if the return code of make is != 0 which is not the case when I run it. Maybe @malter has another idea though.

When minimizing the dual function in REPS / C-REPS, why is the gradient approximated numerically rather than using the analytical expressions provided here? Is this for numerical stability?

The next release will be compatible with Windows and MacOS

• Decide if the unified config will be part of the release -> 0.3
• Update NEWS.md (support for Windows / MacOS)
• Update version number to 0.2.0
• Prepare offline distribution
• Make release on Github
• Upload to mloss.org
• Upload new docker image
• Update docs

dmp.imitate throws error if alpha==0.0 and n_steps < n_params

• env.sh should only contain paths based on $AUTOPROJ_CURRENT_ROOT (https://github.com/rock-simulation/pybob/blob/master/environment.py#L87). This should be set the first time you run env.sh.
• bolero-dev/pybob/pybob.yml must be updated

This is certainly related to

#define private public

in the test:

[ 64%] Building CXX object test/CMakeFiles/dmp_cpp_rigid_body_test.dir/test_rigid_body_dmp.cpp.o
In file included from /usr/include/c++/5/complex:45:0,
                 from /usr/local/include/eigen3/Eigen/Core:28,
                 from /usr/local/include/eigen3/Eigen/Geometry:4,
                 from /home/afabisch/Projekte/rock-bolero/learning/bolero/src/representation/dmp/src/QuaternionDmp.h:2,
                 from /home/afabisch/Projekte/rock-bolero/learning/bolero/src/representation/dmp/src/RigidBodyDmp.h:2,
                 from /home/afabisch/Projekte/rock-bolero/learning/bolero/src/representation/dmp/test/test_rigid_body_dmp.cpp:8:
/usr/include/c++/5/sstream:300:7: error: ‘struct std::__cxx11::basic_stringbuf<_CharT, _Traits, _Alloc>::__xfer_bufptrs’ redeclared with different access
       struct __xfer_bufptrs

when I run ./bootstrap_bolero.sh to install bolero, it went into error while building dmp.

Sine it did not print the reason why this failed, I can't do anything to fix this.

Below is what it looks like:

Install: ['learning/bolero/bolero/wrapper', 'learning/bolero/src/controller', 'learning/bolero/src/representation/dmp']
learning/bolero/src/controller configured
learning/bolero/bolero/wrapper configured
learning/bolero/bolero/wrapper installed
learning/bolero/src/representation/dmp configured
learning/bolero/src/controller installed
learning/bolero/src/representation/dmp build error
Install: ['learning/bolero']
learning/bolero configured
learning/bolero build error

Errors:

• build: learning/bolero/src/representation/dmp
• build: learning/bolero

these dependencies are not handled correctly in the installation process at the moment:

• cython - has to be installed from pip, we need a recent version (?)

python-sklearn is dep from: learning/bolero
python-matplotlib is dep from: learning/bolero
python-scipy is dep from: learning/bolero

Under python3 the import from ._wrapper import CppBLLoader in bolero/wrapper/__init__.py fails.
Therefore I can't create a controller instance from a dict. The config works for bolero_controller

from bolero.controller import Controller
cfg = yaml.load(open("../learning_config.yml"))
Controller(cfg)

Did I miss to install something?
exported PYTHON and CYTHON to version 3, did cmake .., make install in build, also did python3 setup.py install

This must be a scaling problem. Variance 1 seems to work, everything else breaks.

• some parameters are given as environment variables (BL_SEED, BL_LOG_PATH, BL_CONF_PATH), some are read from learning_config.yml
• some environments don't use BL_CONF_PATH: CEC13, FunctionApproximation, MarsEnvironment / this will be addressed with #41
• there is commented code in MountainCar
• ... and in the Controller
• there is no behavior search in C++
• sometimes it would be good to use smart pointers

missing:

• Dict
• MultiBinary
• MultiDiscrete
• Tuple

I added some unit tests but there are still some things missing, for example, reading and saving configuration files.