A general algorithm for eliminating critical conditions for solving the problem of controlling a real walking robot based on deep reinforcement learning methods

The object of the study is a mobile walking robot with two or more movable articulated limbs. The concept of a "critical condition" is introduced, in which the mechanism balances on the verge of falling (but does not fall) or there is a possibility of damage to mechanical components due to the generation of unacceptable joint angles. The subject of the study is a general algorithm for the elimination of critical conditions, which provides the possibility of training an agent based on a deep reinforcement learning algorithm directly on a real robot, without the risk of damaging its mechanisms and interrupting the process of interaction with the environment to restore a stable state. The purpose of this work is to develop a general algorithm for the elimination of critical conditions in the context of adaptive control of a walking robot based on deep learning algorithms with reinforcement. A comparison was made between the proposed and standard methods of applying deep OP on a real robot. The experiments were conducted on 6,000 episodes, with a dimension of 300 steps each. The following quality metrics were selected for evaluation: the percentage of episodes without an actual fall, the percentage of fully completed episodes, and the maximum episode length. The algorithm is based on the concept of "critical condition" and uses the following principles and methods: the "trial and error" method, the feedback principle, holding the projection of the center of gravity point in the area of the polygon formed by the points of contact of the limbs with the work surface, which ensures the balancing of the structure and allows you to determine the boundary areas in which the robot is still stable. The scientific novelty of the work lies in the proposed approach, which allows an intelligent agent to control a physical robot "directly", without pre-configuration in a simulation environment with subsequent transfer implementation. The proposed algorithm is not aimed at improving the agent's performance, but is intended to provide greater autonomy in the learning process of the robot, directly in the hardware. The basic idea is to immediately respond to a critical condition in the form of the fastest sequential return to a certain number of steps back along the decision-making trajectory, ensuring that the agent remains in a stable and safe state at all times. The method of proximal policy optimization (PPO) was used as a method of deep reinforcement learning. As a result of the comparative analysis, the proposed algorithm demonstrated a hundredfold increase in the stability of the mechanism.