Learning to Control Inverted Pendulum on Surface with Free Angles Through Policy Distillation

Research on the inverted pendulum has played an important role in promoting the development of balance control, due to its nonlinear and unstable characteristics. However, existing research ignored the problem of balance control on irregular surfaces, which is a common scenario in real world control, such as a bipedal robot walking on an irregular surface, or a robotic arm performing mobile grasping. In this paper, we propose a two stage control strategy based deep reinforcement learning and policy distillation, named Two-Stage Policy Distillation (TSPD), to control the inverted pendulum on irregular surfaces. In detail, we first construct a general dynamic model of an inverted pendulum that can model with free angles. Following this, we found that by introducing our policy distillation method, which transfers the knowledge gained from inclined surface training to irregular surface training, leads to more stable control instead of directly training the inverted pendulum on an irregular surface. Furthermore, we divide the transfer task into two phases and apply this method twice, using inclined surfaces as the bridge. The results demonstrate that the policy distillation method is feasible to transfer knowledge from inclined surface to irregular surface. It also shows that the TSPD could significantly improve the control performance.