Multi-Modal Vision at the Edge: Toward Low-Latency Perception for Autonomous Systems

Multi-modal perception is critical for robust and reliable object detection in autonomous systems, particularly under challenging environmental conditions. This paper presents an extended YOLOv8 framework that integrates RGB, LiDAR-based depth, and grayscale modalities using early, mid, and late fusion strategies, specifically tailored for deployment on edge AI platforms. To enable real-time processing, we propose a parallelized depth generation pipeline that efficiently converts sparse LiDAR point clouds into dense depth maps. We evaluate the proposed models on the KITTI dataset and benchmark their accuracy and latency across three edge devices: Raspberry Pi 5, NVIDIA Jetson Nano, and Google Coral Edge TPU. Experimental results demonstrate that while late fusion with all three modalities yields the highest detection accuracy (mAP@50: 92.7%, mAP@50-95: 70.6%), early fusion offers the lowest inference latency. Our findings underscore the trade-off between accuracy and latency, highlighting the need to choose a fusion strategy that aligns with the application requirements.