Integrated Whole-Body Control and Manipulation Method Based on Teacher–Student Perception Information Consistency

In emergency scenarios, we focus on studying how to manipulate legged robot dogs equipped with robotic arms to move and operate in a small space, known as legged emergency manipulation. Although the legs of the robotic dog are mainly used for movement, we found that implementing a whole-body control strategy can enhance its operational capabilities. This means that the robotic dog’s legs and mechanical arms can be synchronously controlled, thus expanding its working range and mobility, allowing it to flexibly enter and exit small spaces. To this end, we propose a framework that can utilize visual information to provide feedback for whole-body control. Our method combines low-level and high-level strategies: the low-level strategy utilizes all degrees of freedom to accurately track the body movement speed of the robotic dog and the position of the end effector of the robotic arm; the advanced strategy is based on visual input, intelligently planning the optimal moving speed and end effector position. At the same time, considering the uncertainty of visual guidance, we integrate fully supervised learning into the advanced strategy to construct a teacher network and use it as a benchmark network for training the student network. We have rigorously trained these two levels of strategies in a simulated environment, and through a series of extensive simulation validations, we have demonstrated that our method has significant improvements over baseline methods in moving various objects in a small space, facing different configurations and different target objects.