MTSF: Multi-Scale Temporal-Spatial Fusion Network for Driver Attention Prediction

Recently, there have been certain important advances in the field of driver attention prediction in dynamic scenarios for the bionic perception of intelligent vehicles. Given the visual characteristics of experienced and skilled drivers, who can rapidly and accurately perceive their environment and effectively identify the significance of environmental stimuli, it is believed that an effective driver attention prediction model should not only extract temporal-spatial features comprehensively but also flexibly highlight the importance of these features. In light of this challenge, this paper proposing an improved multi-scale temporal-spatial fusion network model, which adopts an encoder-fusion-decoder architecture and can fully use the scale, spatial, and temporal information in video. First, two independent feature extraction backbones, one 2D and another 3D, are used to extract four temporal-spatial features of different scales from the input video clip and align them in the feature dimension. Then, features from different levels are added to the channel and fused using an attention mechanism to achieve the 3D-2D soft combination effect guided by spatial features. Finally, hierarchical decoding and prediction are performed on temporal-spatial features of different branches, and the results of multiple branches are fused to generate saliency maps. Experiments on three challenging datasets show that the proposed method is superior to the state- of-the-art methods regarding several saliency evaluation metrics and can predict driver attention more accurately. By using an effective temporal-spatial fusion strategy based on attention mechanism, the proposed method can detect important targets and identify risk areas for a human-like autonomous driving system.