Depth-Aware Multi-Task Deep Learning Approach for Railway Scene Perception

Reliable real-time sensing in complex scenes is crucial for safe and autonomous operation in intelligent rail systems. This study proposes a multi-task and depth-aware sensing system that combines semantic segmentation, object detection, and stereo-based depth estimation for railway environments. The proposed system uses YOLOv8–YOLOv12 object detection models and UNet, DeepLabV3+, FPN, and PSPNet-based segmentation architectures. All learning-based segmentation and detection models were trained on the RailSem19 dataset. Experimental results showed that YOLOv9 and YOLOv11 provided stable performance in object detection, while DeepLabV3+ achieved better results in semantic segmentation. To improve system performance, two multi-task fusion approaches were investigated. The first is a segmentation-based geometric inference approach. The second is a dual-stream fusion architecture combining DeepLabV3+ and YOLOv11. The results showed that the dual-stream structure provides higher accuracy and better spatial consistency in railway scenes with occlusions, complex infrastructure, and moving objects. With the ZED2 stereo camera, 2D detection is supported by 3D spatial analysis. An accuracy of ±0.35 m is achieved in tasks such as rail intrusion detection and distance-dependent risk assessment. Overall, the proposed multimodal detection architecture offers a robust and viable solution for railway safety and autonomous monitoring applications.