MSTrack: Multi-Modal Feature Aggregation and Spatio-Temporal Memory Context Encoding for Ground and UAV RGB-Thermal Infrared Tracking

RGB-thermal infrared (RGBT) target tracking integrates spatial features of both RGB and thermal infrared (TIR) modalities, providing a discriminative basis for target feature representation and state estimation. However, existing methods suffer from insufficient cross-modal feature interaction and inadequate spatio-temporal contextual correlation, which limits tracking stability and accuracy in challenging scenarios. To address these issues, we propose MSTrack, a novel tracker that integrates a multi-modal feature aggregation (MMFA) module and a spatio-temporal memory context encoding (STMCE) module. First, to capture both shared features and complementary details across modalities, we design the MMFA module, which employs parallel Swin Transformers and four cross-modal feature interaction (CMFI) components to achieve progressive fusion of RGB and TIR features. Specifically, CMFI integrates a cross-modal collaborative attention mechanism, a triplet attention mechanism, a bidirectional cross-modal weighted fusion mechanism, and a multi-stage information aggregation mechanism to achieve cross-modal interactions. Second, to fuse the key spatio-temporal features across multiple frames, we design the STMCE module. This STMCE module leverages a historical feature memory pool (HFMP) and a spatio-temporal context encoder (STCE) to model global dependencies among the template, current, and historical frames. Finally, the prediction head outputs the target bounding box. Experiments on four ground-based RGBT datasets demonstrate that the proposed tracker outperforms other state-of-the-art trackers. Furthermore, the results on the UAV-based VTUAV dataset validate its generalization potential. Specifically, MSTrack achieves PR/SR scores of 0.877/0.637 on RGBT234 and 0.669/0.530 on LasHeR. On the VTUAV dataset, it achieves MPR/MSR scores of 0.812/0.672 on the short-term subset and 0.595/0.501 on the long-term subset.

Authors

Institutions

Publication Details

Journal
Remote Sensing
Published
2026-11-20
DOI
https://doi.org/10.3390/rs18224004
Primary Topic
Video Surveillance and Tracking Methods
Type
article
Field-Weighted Citation Impact
0.00

Funders

Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

MSTrack: Multi-Modal Feature Aggregation and Spatio-Temporal Memory Context Encoding for Ground and UAV RGB-Thermal Infrared Tracking

Pei Xiang, Jiangluqi Song, Huixin Zhou, Soon Ki Jung et al.
6 links
Remote Sensing
Video Surveillance and Tracking Methods
article

MSTrack: Multi-Modal Feature Aggregation and Spatio-Temporal Memory Context Encoding for Ground and UAV RGB-Thermal Infrared Tracking

Pei Xiang, Jiangluqi Song, Huixin Zhou, Soon Ki Jung, Jin Shi, Xiang Teng, Shuxia Qi
article en

Abstract

RGB-thermal infrared (RGBT) target tracking integrates spatial features of both RGB and thermal infrared (TIR) modalities, providing a discriminative basis for target feature representation and state estimation. However, existing methods suffer from insufficient cross-modal feature interaction and inadequate spatio-temporal contextual correlation, which limits tracking stability and accuracy in challenging scenarios. To address these issues, we propose MSTrack, a novel tracker that integrates a multi-modal feature aggregation (MMFA) module and a spatio-temporal memory context encoding (STMCE) module. First, to capture both shared features and complementary details across modalities, we design the MMFA module, which employs parallel Swin Transformers and four cross-modal feature interaction (CMFI) components to achieve progressive fusion of RGB and TIR features. Specifically, CMFI integrates a cross-modal collaborative attention mechanism, a triplet attention mechanism, a bidirectional cross-modal weighted fusion mechanism, and a multi-stage information aggregation mechanism to achieve cross-modal interactions. Second, to fuse the key spatio-temporal features across multiple frames, we design the STMCE module. This STMCE module leverages a historical feature memory pool (HFMP) and a spatio-temporal context encoder (STCE) to model global dependencies among the template, current, and historical frames. Finally, the prediction head outputs the target bounding box. Experiments on four ground-based RGBT datasets demonstrate that the proposed tracker outperforms other state-of-the-art trackers. Furthermore, the results on the UAV-based VTUAV dataset validate its generalization potential. Specifically, MSTrack achieves PR/SR scores of 0.877/0.637 on RGBT234 and 0.669/0.530 on LasHeR. On the VTUAV dataset, it achieves MPR/MSR scores of 0.812/0.672 on the short-term subset and 0.595/0.501 on the long-term subset.

Remote SensingVol. 18(22)
Xidian University (CN), Kyungpook National University (KR)
National Natural Science Foundation of China, China Postdoctoral Science Foundation, Higher Education Discipline Innovation Project
Video Surveillance and Tracking Methods
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon