Designing standard library of manipulation skill-agents for Learning-from-Observation

To use new robot hardware, it is necessary to develop a control program tailored to the specific robot. Considering the reusability of software among robots is crucial for minimizing the effort involved in this process and maximizing software reuse across different robots. This paper proposes a method to reduce the effort of software development for a specific robot by considering hardware-level reusability, using a Learning-from-Observation (LfO) framework with a pre-designed skill library. The LfO framework first represents the demonstrated actions in hardware-independent representations, referred to as task models, from observing human demonstrations, and captures the necessary parameters for the interaction between the environment and the robot (Ikeuchi et al., 2024). Then, for executing the demonstrated actions, a set of skill agents is employed to convert the representations of the task models into robot commands. This paper focuses on the latter part of the LfO framework and explores a hardware-independent design approach for these skill agents. Especially, we dedicate a manipulation skill library. To achieve this, we describe these skill agents in a hardware-independent manner based on the physical characteristics of a grasped object and an environment. This paper, first, defines a necessary and sufficient skill-agent set corresponding to covering all possible actions, and considers the design principles for these skill agents. We provide concrete examples of such skill agents and demonstrate the practicality of these skill agents by showing that the same representations can be executed on two different robots, Nextage and Fetch, and two different end-effectors, Shadow Hand-Lite and parallel gripper.