Multi-objective path integral policy improvement for learning robotic motion

Abstract This paper proposes a new multi-objective reinforcement learning (MORL) algorithm for robotics by extending policy improvement with path integral ( $$\text {PI}^2$$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"> <mml:msup> <mml:mtext>PI</mml:mtext> <mml:mn>2</mml:mn> </mml:msup> </mml:math> ) algorithm. For a robot motion acquisition problem, most existing MORL algorithms are hard to apply, because of the high-dimensional and continuous state and action spaces. However, policy-based algorithms such as $$\text {PI}^2$$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"> <mml:msup> <mml:mtext>PI</mml:mtext> <mml:mn>2</mml:mn> </mml:msup> </mml:math> can be applied to solve this problem in single-objective cases. Based on the similarity of $$\text {PI}^2$$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"> <mml:msup> <mml:mtext>PI</mml:mtext> <mml:mn>2</mml:mn> </mml:msup> </mml:math> and evolution strategies (ESs) and the fact that ESs are well-suited for multi-objective optimization, we propose an extension of $$\text {PI}^2$$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"> <mml:msup> <mml:mtext>PI</mml:mtext> <mml:mn>2</mml:mn> </mml:msup> </mml:math> and some techniques to speed up the learning. The effectiveness is shown via numerical simulations.