Q-HIRL: A Q-Network Representation-Based Request Framework for Human-in-the-Loop Reinforcement Learning

Human-in-the-loop reinforcement learning (HIRL) has emerged as a promising approach to address the challenges of sample efficiency and exploration in complex environments. This paper introduces a novel HIRL method called Q-HIRL (Q-Network Representation-Based Human-in-the-Loop Reinforcement Learning), which leverages Q-network representations to guide expert interventions and enhance learning efficiency. Unlike traditional methods that rely solely on Q-value differences, Q-HIRL utilizes the similarity between Q-network representations to determine when to request human guidance, resulting in more stable and efficient learning. The proposed approach dynamically adjusts the timing of expert interventions based on the convergence of representation similarity, adapting to different environments and improving overall robustness. Extensive experiments conducted across five MuJoCo environments demonstrate that Q-HIRL outperforms state-of-the-art HIRL methods and traditional reinforcement learning algorithms in terms of convergence speed and sample efficiency. Notably, Q-HIRL achieves superior results while requiring fewer human expert interventions compared to existing approaches.