TSFormer: Tracking Structure Transformer for Image Inpainting

Recent studies have shown that image structure can significantly facilitate image inpainting. However, current approaches mostly explore structure prior without considering its guidance to texture reconstruction, leading to performance degradation. To solve this issue, we propose a two-stream Tracking Structure Transformer (TSFormer), including structure target stream and image completion stream, to capture the synchronous and dynamic interplay between structure and texture. Specifically, we first design a structure enhancement module to restore the Histograms of Oriented Gradient (HOG) and the edge of an input image in a sketch space, which forms the input of the structure target stream. Meanwhile, in the image completion stream, we design a channel-space parallel-attention component to facilitate the efficient co-learning of channel and spatial visual cues. To build a bridge between the two streams, we further develop a structure-texture cross-attention module, wherein both structure and texture are synchronously extracted through self-attention, and texture extraction is implemented by dynamically tracking the structure in a cross-attention fashion, enabling the capture of the intricate interaction between structure and texture. Extensive experiments evaluated on three benchmark datasets, including CelebA, Places2, and Paris StreetView, demonstrate that the proposed TSFormer achieves state-of-the-art performance compared to its competitors. The code is available at https://github.com/GZHU-DVL/TSFormer .