Memory-Based Temporal Transformer U-Net for Multi-Frame Infrared Small Target Detection

In the field of infrared small target detection (ISTD), single-frame ISTD (SISTD), using only spatial features, cannot deal well with dim targets in cluttered backgrounds. In contrast, multi-frame ISTD (MISTD), utilizing spatio-temporal information from videos, can significantly enhance moving target features and effectively suppress background interference. However, current MISTD algorithms are limited by fixed-size time windows, resulting in an inability to adaptively adjust the input amount of spatio-temporal information for different detection scenarios. Moreover, utilizing spatio-temporal features remains a significant challenge in MISTD, particularly in scenarios involving slow-moving targets and fast-moving backgrounds. To address the above problems, we propose a memory-based temporal Transformer U-Net (MTTU-Net), which integrates a memory-based temporal Transformer module (MTTM) into U-Net. Specifically, MTTM utilizes the proposed D-ConvLSTM to sequentially transmit the temporal information in the form of memory, breaking through the limitation of the time window paradigm. And we propose a Transformer-based interactive fusion approach, which is dominated by spatial features of the to-be-detected frame and supplemented by temporal features in the memory, thereby effectively dealing with targets and backgrounds with various motion states. In addition, MTTM is divided into a temporal channel-cross Transformer module (TCTM) and a temporal space-cross Transformer module (TSTM), which achieve target feature enhancement and global background perception through feature interactive fusion in the channel and space dimensions, respectively. Extensive experiments on IRDST and IDSMT datasets demonstrate that our MTTU-Net outperforms existing MISTD algorithms, and they verify the effectiveness of the proposed modules.