Human-Guided Reinforcement Learning Using Multi Q-Advantage for End-to-End Autonomous Driving

Reinforcement learning (RL) is a promising approach for end-to-end autonomous driving. However, training a RL strategy for autonomous driving is challenging, requiring a meticulously crafted reward function and an extensive number of interactions. This necessitates reconsidering the integration of RL with human guidance, acknowledging the potential benefits of combining machine learning with human expertise. Herein, we propose a novel human-guided reinforcement learning (HGRL) algorithm for policy training in the context of end-to-end autonomous driving. Our HGRL method employs several mechanisms to enhance the effectiveness of low-intensity human guidance, including a regressed human policy model, multiple Q-advantage technique, and prioritized human experience replay. The proposed method is evaluated on a challenging lane-changing and overtaking driving task based only on small neural networks and image inputs. Simulation results show that our method surpasses current state-of-the-art human-guided RL algorithms in both driving performance and generalization performance. Furthermore, the proposed HGRL method trained in simulation is then transferred to a real-world autonomous vehicle.