Development of knowledge-selective transfer reinforcement learning with heterogeneous-domain knowledge sets

Abstract In this study, we propose a knowledge-selective transfer reinforcement learning method that simultaneously achieves heterogeneous domain transfer and knowledge selection in autonomous robots. In recent years, autonomous robots capable of recognition, decision, and action in their environments have been developed, and their utilization is advancing in a wide range of fields such as disaster response and logistics support. Reinforcement learning (RL) enables adaptive learning in unknown environments and is instrumental in realizing technologies such as autonomous robots. However, RL requires extensive exploration, leading to the issue of long training times. To address this, transfer reinforcement learning (TRL) has been introduced to reduce the training time by reusing previously learned knowledge. Nevertheless, the transfer effectiveness depends on the knowledge reused, creating a need for appropriate knowledge selection. Therefore, we propose heterogeneous domain SAP-net (HDSAP-net), which enables knowledge transfer by utilizing heterogeneous-domain knowledge sets acquired from various agents and tasks, thus extending the existing Spreading Activation Policy Network (SAP-net). In its algorithm design, HDSAP-net incorporates inter-task mapping using linear interpolation to bridge discrepancies between the heterogeneous domains. Furthermore, by analyzing the behavior of HDSAP-net, we formulated a network design method using optimal transport cost and implemented a new activation-value control method, thereby improving its performance. This enables TRL, wherein knowledge sets derived from various robots are shared among various agents, which was challenging with the conventional SAP-net. Verification experiments using the physics simulator Webots involving a mobile robot, robotic arm, and drone demonstrated that HDSAP-net significantly improves the learning efficiency when compared with that of conventional RL. The proposed method reduced the learning time by 32.1% to 82.0%, confirming its capability of autonomously discovering and utilizing effective knowledge across various domains.