DSTA-Net: Deformable Spatio-Temporal Attention Network for Video Inpainting

Video inpainting aims to fill in the corrupted or missing regions of target frame with plausible contents by exploiting the information drawn from surrounding reference frames. However, existing methods always treat reference frames uniformly and assign equal importance to all reference frames, which will lead to inpainting results that are too smooth to preserve necessary details and textures. For this purpose, we design a Deformable Spatio-Temporal Attention (DSTA) network, named DSTA-Net, to automatically learn to pay higher attention on specific reference frames. Concretely, our DSTA-Net begins by employing a lightweight motion estimator to predict coarse optical flows between target frame and reference frames. These coarse optical flows are employed as a basic offset rather than directly aligning reference frames. Subsequently, our proposed DSTA module calculates the sampling point offsets, attention maps, and flatten features with the aim of capturing the appropriate pixels from the regions with the highest similarity to the reference frames to generate the corrupted or missing contents. Numerous experimental results show that our method significantly outperform recent seven benchmarks on two most commonly-used datasets.