Multi-head deep Q-learning for continuous beamforming with selective MC-CDMA operation in V2X highway communications

This study investigates a large-scale dynamic Vehicle-to-Everything (V2X) communication network, in which multiple Roadside Units (RSUs) are deployed along highways to enable high-speed vehicular links. To ensure robust and adaptive performance under fast-varying conditions, we propose an integrated framework that combines resource block-based MC-CDMA modulation with dynamic beamforming optimized for complex propagation environments. A custom code mapper and resource element (RE) allocator are introduced to support interference-aware transmission and enhance signal robustness in dense deployment scenarios. The MC-CDMA scheme enables extended-range coverage per RSU, outperforming traditional OFDM-based transmission in terms of reliability and scalability. To further optimize performance, a Deep Reinforcement Learning (DRL) model is employed to jointly handle beam tracking and time-varying channel conditions. Specifically, a physics-inspired Deep Q-Learning (DQL) strategy is proposed, using a force-arm-based mechanism to adaptively correct beam misalignment caused by mobility and Doppler effects. Simulation results demonstrate that the proposed system achieves significant improvements in bit error rate (BER), bitrate stability, handover smoothness, and spectral efficiency. When equipped with a large-scale antenna array, the system ensures continuous beam tracking and substantially outperforms conventional RL-based techniques. These results highlight its potential for future 6G-enabled V2X deployments, where scalability, adaptability, and robust link quality are essential.