Towards Comprehensive Roadside Intelligence: Sensor Fusion and Full-Stack Perception with Multiple Cameras

Roadside perception has become a critical component for connected and automated vehicles (CAVs), enhancing safety and offering a comprehensive view of the traffic environment that onboard detection systems alone cannot provide. By supplementing the limitations of onboard sensors, roadside perception systems improve the accuracy and reliability of detecting and localizing vehicles and pedestrians in challenging locations. Currently, a variety of cameras, including fisheye and regular cameras, are deployed along roadsides for surveillance purposes. These sensors have significant potential to improve vehicle and pedestrian detection. This paper extends our previous work on single image sensor vehicle detection by developing a comprehensive multiple sensor fusion framework. We take advantage of the complementary strengths of multiple fisheye and regular cameras to enhance the accuracy and robustness of the perception system. The proposed system has been extensively tested in Mcity, a controlled urban testing environment, through numerous field tests. The results demonstrate the effectiveness of our approach, showcasing promising improvements in vehicle and pedestrian detection and tracking accuracy.