Real-time railroad crossing surveillance with edge computing

Accurate and effective detection of anomalies at rail crossings is critical for improving railway safety. However, existing methods often struggle to reliably identify intrusions and threats. To address these challenges, this study introduces You Only Look Once-Region-based Convolutional Neural Network (YOLO-RCNN), a hybrid framework that incorporates YOLO-FG (foreground), a component specifically designed to detect and segment all objects within a scene, capabilities that conventional object detectors lack. The proposed model further integrates an RCNN with the Region of Interest Align (RoIAlign) mechanism, effectively classifying and tracking detected objects. This framework enables precise detecting, classifying, and tracking objects with high computational efficiency, and effectively overcomes the limitations of traditional “classification and tracking by detection” pipelines, significantly reducing computational overhead. Optimized for real-time applications on resource-constrained edge devices, YOLO-RCNN achieves an F-measure of 90.49 % on the Change Detection Network (CDnet) 2014 dataset and demonstrates its effectiveness on the custom Railroad Crossing Dataset (RCD), achieving a mean average precision (mAP) of 54.15 %, a Seg mAP of 43.62 %, and a Higher Order Tracking Accuracy (HOTA) score of 63.64 %. Deployment optimizations using TensorRT and oneAPI Threading Building Blocks (oneTBB) increased inference speed from 4.69 Frames Per Second (FPS) to 54.79 FPS on desktop systems and from 3.39 FPS to 29.19 FPS on Jetson AGX Orin, demonstrating its real-time applicability. These results underscore YOLO-RCNN's potential as a robust solution for efficient and reliable railroad crossing monitoring, as well as other real-time surveillance tasks. The RCD dataset is publicly released at: https://www.kaggle.com/datasets/elvin1233/rail-crossing .