An Embedded System Architecture for Robust Deep Learning-Based Traffic Light Detection

Deployment of deep learning (DL)-based automotive systems must account for sensor failures, which can occur naturally or artificially and severely degrade DL model performance. While industrial adoption of automated driving (AD) technologies advances, research on sensor vulnerabilities in relation to DL-based perception remains limited – a critical gap given the need for highly dependable systems. We address this challenge through two key contributions in the development of an industrial-strength Traffic Light Detection (TLD) system. First, we evaluate the impact of camera sensor vulnerabilities on the state-of-the-art YOLO11 object detector using both public and proprietary datasets. Our results reveal significant performance degradation under common sensor issues, such as adverse weather conditions and damaged image stabilizers. Second, we propose a system architecture designed to: detect sensor failures, clean corrupted frames, perform TLD when feasible, filter out unrecoverable frames, and trigger a Takeover Request (TOR) if failures persist. By decoupling noise mitigation from standard object detection, our approach enhances both performance and extensibility. The system is implemented on an embedded platform (NVIDIA Jetson Orin Nano) and evaluated using simulated sensor failure scenarios on a short-trip vehicle video. Results demonstrate robust handling of sensor failures with minimal performance degradation and limited TOR rates. Our analysis highlights critical challenges and necessary improvements for effective real-world deployment.