Monitoring automotive perception sensors using latent representations

Abstract Safeguarding autonomous vehicles is a constant challenge, since unknown circumstances that the system may not be able to handle can always arise in real-world traffic. This work proposes a monitoring framework for automotive perception sensors to detect such situations. The objective is to detect anomalous behavior from LiDAR and camera sensors at the level of object state estimations. A contrastive embedding method is used to map object states into a structured latent space. An intelligent trigger utilizes this representation space to perform anomaly detection. A key feature of the monitoring framework is that no anomaly labels are required during the training. Further, the proposed monitoring framework can be applied online, complying with ISO 21448 regarding operation phase activities. Experiments are performed on the publicly available real-world nuScenes dataset.