Revisiting Performance Metrics for Multimodal mmWave Beam Prediction Using Deep Learning

Directional millimeter-wave (mmWave) wireless communication systems with large antenna arrays face challenges in dynamically managing beams for mobile users. Recently, integrating sensors into communication frameworks has garnered interest for enhancing situational awareness. This paper introduces a novel end-to-end multimodal deep learning architecture for mmWave beam prediction. Our methodology incorporates multiple sensors, including camera, RADAR, LiDAR, and GPS, to improve beam prediction accuracy and efficiency across different scenarios. We conduct extensive comparisons between single and multimodal approaches, exploring various fusion methods such as early and late fusion. The proposed architecture employs both spatial and temporal feature correlation on the training dataset. Additionally, the current beam prediction performance evaluation metrics assume there is only one correct beam, but the nature of the communication problem suggests thatmultiple beams often perform similarly. Therefore, we propose new metrics that consider multiple beams as valid options and assess performance based on the beam strength. Our results indicate that our early fusion of features from multiple modalities consistently outperforms our late fusion models.