Collision Scenario Analysis for Autonomous Vehicles Using Multimodal Deep Learning Models

As autonomous vehicle technologies continue to advance, there is a critical need for safety testing methods to analyze collisions in different scenarios accurately. However, existing methods are insufficient in collision reasoning because of the diversity of real-world scenarios. This paper utilizes publicly available datasets to analyze essential points of collisions, such as collision causes, responsible parties, and vehicle damage levels. We extracted three key components from collision videos using three deep-learning models, including road surface condition recognition, time-of-day classification, and weather condition detection. Next, a large language model powered by LLaMA 3 is used to analyze extracted information and video transcripts to conduct an individualized diagnosis of each collision case. The effectiveness of the proposed methods is validated through comprehensive experiments across different collision scenarios, demonstrating significant improvements in detection accuracy and analytical depth.