Uni–MPedFormer: towards a unified transformer–based framework for multispectral pedestrian detection

Robust pedestrian detection in real-world scenes remains difficult because of scale variation, body occlusion, and degraded nighttime visibility. RGB-T image pairs offer complementary spectral cues that can mitigate these problems, but most existing methods address each challenge in isolation rather than within a single architecture. We introduce Uni-MPedFormer , a uni fied trans former -based m ultispectral ped estrian detection framework that handles all three modality-shared difficulties within an end-to-end trainable network. The framework consists of three components. First, a Cross-modality Information Fusion Transformer (CIFT) encoder uses the global receptive field of self-attention to capture intra- and inter-modal dependencies between RGB and thermal streams. A Feature Correlation Recalibration (FCR) inner-attention block then enforces neighbourhood-consistent query–key correlations and suppresses spurious responses. Second, a U-shaped Feature Aggregation (UFA) decoder merges multi-scale hierarchical representations through lateral, top-down, and bottom-up pathways, sharpening spatial localisation across pedestrian scales. Third, a lightweight Illumination-aware Weight Generation (IWG) module estimates scene illumination and dynamically re-weights each modality’s contribution to the final detection score. Experiments on five public benchmarks—KAIST, CVC-14, FLIR-ADAS, LLVIP, and UTokyo—and comparisons with recent multimodal detectors show that Uni-MPedFormer achieves competitive or superior accuracy while maintaining a favourable accuracy–speed trade-off (25 FPS on a single 2080Ti). We further report an explicit complexity analysis (parameters, FLOPs, peak memory), a scalability study at higher input resolutions, a sensitivity study on the FCR reduced dimension D , a robustness analysis under simulated RGB-T misalignment, and an explicit module-to-challenge correspondence between CIFT and occlusion, UFA and small-scale pedestrians, and IWG and nighttime detection.