PBO (policy-based optimization) is a degenerate policy gradient algorithm used for black-box optimization. It shares common traits with both DRL (deep reinforcement learning) policy gradient methods, and ES (evolution strategies) techniques. In this repository, we present a parallel PBO algorithm with full covariance matrix adaptation, along with a few demonstrative applications. The related pre-print can be found here and the formal paper here. This paper formalizes the approach used in previous related works:

• Direct shape optimization through deep reinforcement learning (paper, pre-print and github repository),
• Single-step deep reinforcement learning for open-loop control of laminar and turbulent flows (paper and pre-print),
• Deep reinforcement learning for the control of conjugate heat transfer with application to workpiece cooling (paper and pre-print)

After cloning the package, just cd inside the folder and install using:

pip install -e .

The environments from the paper are available in the envs/* folder. For each .py environment file, you need a .json parameter file located in the same directory. To run an environment, just use:

pbo path/to/envs/my_env.json

Below are some selected optimization cases performed with the algorithm.

Simple minimum-finding on textbook analytical functions (see more in the paper).

We consider the minimization on a parabola defined in [-5,5]x[-5,5]. Below is the course of a single run, generation after generation, with a starting point in [2.5,2.5]:

The Rosenbrock function is here defined in [-2,2]x[-2,2]. It contains a very narrow valley, with a minimum in [1,1]. The shape of the valley makes it a hard optimization problem for many algorithms. Here is the course of a single run, generation after generation, with a starting point in [0.0,-1.0]:

Optimal packing of geometrical shapes (see here for much, much more). The goal is to pack unit shapes in the square of smallest side. Most solutions are already known.

Environment	Description
	6 circles in square (12 degrees of freedom): the optimal value is s = 5.328+, best PBO result was s = 5.331 after 800 generations.
	10 circles in square (20 degrees of freedom): the optimal value is s = 6.747+, best PBO result was s = 6.754 after 1000 generations
	3 equilateral triangles in square (9 degrees of freedom): the optimal value is s = 1.478+, best PBO result was s = 1.478+ after 1000 generations
	5 equilateral triangles in square (15 degrees of freedom): the optimal value is s = 1.803+, best PBO result was s = 1.807+ after 2500 generations

We consider the equations of the Lorenz attractor with a velocity-based control term:

We make use of the following non-linear control with four free parameters:

Two control cases are designed: the first one consists in forcing the system to stay in the x<0 quadrant, while the second one consists in maximizing the number of sign changes (cases inspired from this thesis). Below is a comparison between the two controlled cases.