Progressive Transformer With Multimodality Adaptation for RGB–T Tracking

With the continuous advancement of sensor technologies, the combined use of RGB images and thermal infrared images has become a critical approach for improving tracking performance by generating powerful fusion features. However, the inherent spatial-spectral discrepancies between RGB and thermal modalities creates feature distribution mismatches that hinder effective cross-modality alignment, while their distinct focus regions introduce cross-modality semantic conflicts during fusion. To tackle this issue, we propose a Progressive Transformer network with Multi-modality Adaptation (PTrMA) to perform cross-modality interaction for visible-thermal tracking. Specifically, we introduce a progressive transformer network consisting of shallow interaction and deep interaction. The shallow interaction enables coarse cross-modality information transmission through adaptive channel exchange guided by channel correlations, in a bid to maintain the semantic consistency of multiple modalities and reduce the cross-modality distribution gap. Cross-modality features processed by shallow interaction are propagated into deep interaction to perform modality adaptation, where a coupling-decoupling module is designed to efficiently enhance modality-aware complementary information under the mutual interaction of multiple multi-modal images. Finally, the obtained features undergo further modality integration through shallow interaction to enhance semantic consistency for prediction, further improving the tracking performance. This study contributes an innovative fusion method that advances the application of multi sensors in tracking and monitoring. We conduct extensive experiments on three widely used tracking benchmarks, showcasing the superior performance of the proposed tracker.