Efficient Maximum Entropy Reinforcement Learning Framework Guided by Expert Policies: An Application in Autonomous Driving

Deep Reinforcement Learning (DRL) has garnered significant attention for its superior adaptability and continuous learning capabilities in the decision control of autonomous driving. However, in complex human-machine hybrid scenarios with high-dimensional continuous state spaces, such as unsignalized intersections, DRL-based autonomous driving decision-making control algorithms exhibit low sample efficiency and struggle to converge. Moreover, designing a reward function that accurately captures the complexities of human-machine interactions is highly challenging; inappropriate reward design can lead to unintended behaviors by the agent, thereby threatening system safety. To address these issues, this study proposes a Contrastive Expert-Guided Maximum Entropy Reinforcement Learning (CE-MERL) framework. This framework trains a discriminator model using both human expert datasets and the interaction experiences of the agent with the environment to identify expert policies. It then constrains the agent’s policy alignment with the expert policy as a cost within the maximum entropy reinforcement learning framework. This not only enables the agent to conduct reasonable and efficient exploration, thereby improving sample efficiency, but also allows it to learn safer, human-like policies even when using only sparse rewards. To ensure that the discriminator can continuously and effectively identify expert policies, we adopt the Vision Transformer (ViT) model as a feature encoder and introduce auxiliary tasks of contrastive learning. This method achieves efficient coupling of state and action features and mitigates overfitting caused by limited expert demonstration data. Simulation results show that in two tasks: unprotected left turn and ramp merging, our proposed CE-MERL method, without the use of shaped rewards, exhibits higher task success rates and sample efficiency compared to baseline algorithms.