Developing Merging Policies for CAVs: A Policy Training Framework Combining Human Experience With Reinforcement Learning

Freeway on-ramp merging is a challenging task for connected and automated vehicles (CAVs) due to the complex traffic environment. The emergence of reinforcement learning (RL) has accelerated the development of driving strategies for CAVs by leveraging its capability to learn from interactions with the environment. However, the difficulty in achieving convergence in the training process poses challenges for applying RL to driving tasks effectively. Moreover, most RL-based merging strategies are developed in highly parameterized and oversimplified simulation environments, resulting in poor performance in real scenarios. To address this, this study introduces a novel merging policy training framework for CAVs, which includes a merging environment reconstruction method and a two-stage policy training approach. Firstly, the empirical merging scenarios are extracted from the NGSIM dataset and augmented using the calibrated Intelligent Driver Model (IDM), creating a more interactive and realistic training environment for the merging CAV. Then, a two-stage training approach is proposed, where the merging CAV first imitates human behavior under sparse rewards in the first stage and is further optimized under dense rewards in the second stage. The experiment results indicate that the proposed method effectively enhances traffic efficiency by reducing the average merging time of CAVs by 3.6 s and increasing the merging speed by 3.6 ft/s, in comparison to Human-Driven Vehicles (HDVs). The sensitivity analysis also validates the effectiveness and robustness of our proposed framework.