Both Objects and Relationships Matter: Cross Coupled Transformer for Railway Scene Perception

Railway scene status perception is the foundation for ensuring the efficient and safe operation of trains. Achieving comprehensive railway scene perception requires not only detecting objects but also understanding the relationships between them. However, current railway scene perception technologies struggle with the latter, while general object and relationship perception methods perform poorly when facing railway scene with significant structural features and small object characteristics. Thus, this paper proposes Rail-former, the first status perception framework specifically designed for railway scene. Rail-former operates based on two key components: the Railway Feature Enhancement Module (RFEM) and the Cross-Coupled Transformer (CCTF). RFEM first leverages a cluster-based normalization evaluation method to condense railway scene structural features in spatial domain. It then applies high-dimensional information compensation to enhance small-object feature representation. The CCTF implements a cross-coupled structure between object decoder and triplet decoder based on a novel interactive attention mechanism, rather than conventional single-decoder or parallel dual-decoder structures, enabling guided enhancement of both object features and relationship features in railway scene. Additionally, we introduce a multimodal matching loss function that incorporates masks, categories, and bounding boxes to mitigate overfitting to a single modality during training. To the best of our knowledge, Rail-former is the first work in the railway domain that leverages panoptic parsing results to reveal relationships. Extensive experiments conducted on both our self-constructed railway dataset and a public dataset demonstrate the superior performance of our approach, achieving accuracy rates of 49.9% and 37.4%, surpassing the best competing methods by 7.8% and 1.1%, respectively.