Towards Multi-Modal Crash Prediction Based on V2X and Visual Information Using a Social Robot

The development of autonomous vehicles and vehicular communications (V2X) promises to significantly enhance traffic safety and efficiency. However, challenges remain in ensuring safe interactions between autonomous vehicles and vulnerable road users (VRUs). We promote the idea to use a social robot as interface between social interaction and V2X. We propose to use such a robot for crash prediction: the social robot, equipped with an RGB-D camera and V2X-communication capabilities, gathers data on pedestrians’ trajectories and vehicles’ movement within a shared environment. The data can then be used to predict possible crashes. In this ongoing work, we present a framework that integrates the basic functionality to implement such an approach. To test the approach a data set consisting of videos of crossing pedestrians and V2X data of an eBike was collected. The system effectively converts the trajectories of pedestrians and vehicles into a shared coordinate frame, enabling precise detection of potential collisions. The preliminary findings show potential for a novel method for crash prediction.