Smart Railway Safety: Integrating Deep Learning with Vision Transformers for Obstacle Detection and Track Health Monitoring

Railway safety is critical for preventing accidents caused by obstacles present on the track and infrastructure failures. Unlike, other monitoring systems that often solely focus on either obstacle detection or track condition analysis, limiting their ability to provide a comprehensive risk assessment. This research paper presents a multi-modal deep learning framework where a real-time obstacle detection was performed with the use of the YOLO model. However, detecting an obstacle alone is not sufficient for assessing the collision risk; therefore, a Vision Transformer was incorporated to further refine the detection. Vision Transformers enable a detailed understanding of the obstacle type and characteristics; thereby assisting in increasing the efficiency of impact evaluation. Simultaneously, the condition of the tracks was analyzed using Mask R-CNN alongside YOLO to detect cracks on sleepers and identify loose or broken fasteners. A custom dataset consisting of one-minute track videos, where each frame was converted into images, was used for model evaluation. This multi-modal approach ensures high precision in identifying obstacles as well as the track condition. To evaluate the outputs from obstacle analysis and track monitoring, a fusion model is used to aggregate the outputs, resulting in a risk- weighted decision score. This score assists in deciding the next possible action for the train to take while operating. The proposed system shows higher accuracy, improves the railway monitoring and minimises accident risks, making it suitable for real-time deployment in railway networks.