A Multi-Agent Cooperative Decision-Making Method with Risk-Aware Dual Masking for Highway On-Ramp Merging

Highway on-ramp merging in mixed-traffic environments is a high-dimensional combinatorial optimization problem that requires joint consideration of driving safety, traffic efficiency, and multi-agent coordination. To address this issue, this paper proposes BQDOM, namely Beam-searched Q-learning with Dual-mask for On-ramp Merging, a joint-action scoring-based multi-agent Q-learning method. The proposed method first constructs historical observation sequences from environmental perception information and uses them to generate trajectory embeddings and short-term future trajectory predictions. The trajectory embeddings enhance individual action-value estimation, while the predicted trajectories support action mask generation. By integrating trajectory embeddings with real-time observations, a shared individual Q-network evaluates the action values of each agent. Meanwhile, future trajectories are used to construct a risk mask, which is combined with a legality mask to form a dual action mask that filters high-risk and invalid actions. A fixed-width Beam Search strategy is then adopted to build candidate joint-action sequences. Key-action weights are assigned according to vehicle spatial positions, and candidate joint actions are iteratively scored. The joint action with the highest score is selected for execution, and the executed joint action is used to construct a system-level temporal-difference target for updating the action-value networks. Experiments in a highway-based simulation environment compare BQDOM with MAA2C, MAPPO, and MAACKTR across the Easy, Normal, and Hard scenarios. BQDOM achieves the best overall performance among the compared methods, reducing the mean collision rate relative to the best baseline for this metric by 75.00%, 22.22%, and 18.18%, respectively. These results suggest a favorable balance between safety and traffic efficiency within the evaluated simulation scenarios.