MSTrack: Multi-Modal Feature Aggregation and Spatio-Temporal Memory Context Encoding for Ground and UAV RGB-Thermal Infrared Tracking
RGB-thermal infrared (RGBT) target tracking integrates spatial features of both RGB and thermal infrared (TIR) modalities, providing a discriminative basis for target feature representation and state estimation. However, existing methods suffer from insufficient cross-modal feature interaction and inadequate spatio-temporal contextual correlation, which limits tracking stability and accuracy in challenging scenarios. To address these issues, we propose MSTrack, a novel tracker that integrates a multi-modal feature aggregation (MMFA) module and a spatio-temporal memory context encoding (STMCE) module. First, to capture both shared features and complementary details across modalities, we design the MMFA module, which employs parallel Swin Transformers and four cross-modal feature interaction (CMFI) components to achieve progressive fusion of RGB and TIR features. Specifically, CMFI integrates a cross-modal collaborative attention mechanism, a triplet attention mechanism, a bidirectional cross-modal weighted fusion mechanism, and a multi-stage information aggregation mechanism to achieve cross-modal interactions. Second, to fuse the key spatio-temporal features across multiple frames, we design the STMCE module. This STMCE module leverages a historical feature memory pool (HFMP) and a spatio-temporal context encoder (STCE) to model global dependencies among the template, current, and historical frames. Finally, the prediction head outputs the target bounding box. Experiments on four ground-based RGBT datasets demonstrate that the proposed tracker outperforms other state-of-the-art trackers. Furthermore, the results on the UAV-based VTUAV dataset validate its generalization potential. Specifically, MSTrack achieves PR/SR scores of 0.877/0.637 on RGBT234 and 0.669/0.530 on LasHeR. On the VTUAV dataset, it achieves MPR/MSR scores of 0.812/0.672 on the short-term subset and 0.595/0.501 on the long-term subset.
Authors
- Pei Xiang (ORCID: https://orcid.org/0000-0003-1895-1894)
- Jiangluqi Song (ORCID: https://orcid.org/0000-0003-1497-4810)
- Huixin Zhou (ORCID: https://orcid.org/0000-0001-5397-5221)
- Soon Ki Jung (ORCID: https://orcid.org/0000-0003-0239-6785)
- Jin Shi (ORCID: https://orcid.org/0009-0000-2015-8735)
- Xiang Teng (ORCID: https://orcid.org/0009-0005-8744-7351)
- Shuxia Qi (ORCID: https://orcid.org/0009-0001-7042-6096)
Institutions
- Xidian University (CN)
- Kyungpook National University (KR)
Publication Details
- Journal
- Remote Sensing
- Published
- 2026-11-20
- DOI
- https://doi.org/10.3390/rs18224004
- Primary Topic
- Video Surveillance and Tracking Methods
- Type
- article
- Field-Weighted Citation Impact
- 0.00
Funders
- National Natural Science Foundation of China
- China Postdoctoral Science Foundation
- Higher Education Discipline Innovation Project