Hierarchical boundary feature alignment network for video salient object detection

The deep learning based video salient object detection (VSOD) models have achieved great success in the past few years, however, these VSOD models still suffer from the following two problems: i) struggle in accurately predicting those pixels surrounding salient objects; ii) unaligned features of different scales lead to deviations in feature fusion . To tackle these problems, we propose a hierarchical boundary feature alignment network (HBFA). Specifically, the proposed HBFA consists of a temporal–spatial fusion module (TSM) and three decoding branches. TSM captures multi-scale spatiotemporal information. The two boundary feature branches are used to guide the whole network to pay more attention to the boundary of salient objects, while the feature alignment branch is capable of fusing the features from the internal and external branches while aligning features across different scales. Our extensive experiments show that the proposed method reaches a new state-of-the-art performance. • We propose a novel architecture to address the boundary blurring issue in video salient object detection tasks. • We developed the TSM module with a dual-stream cross-attention mechanism and a foreground query generator. It reduces computational complexity and integrates appearance and motion features effectively. • We propose an FAM module. It turns the reverse attention mechanism into a three-branch structure with dynamic weight allocation, enhancing spatiotemporal feature alignment and integration across scales. • We validate our proposed method on common benchmarks. Results show it outperforms previous state-of-the-art models across multiple evaluation metrics.