Image inpainting based on decoupled spatiotemporal transformer

To address the limitations of image inpainting algorithms that utilize single-frame and single-view image sequences for degraded image inpainting in robotic scenarios, a novel defaced image inpainting framework for robot binocular vision systems based on a decoupled spatial transformer is proposed. This approach integrates the spatial information of adjacent reference perspectives and the temporal information of image sequences. Initially, an encoder extracts features from the multi-frame data of the current perspective, with a decoupled temporal transformer module retrieving information that may be occluded or degraded in the current frame but is visible in the reference frame, as well as the usable information in the current frame. Then, a correlation computation layer within the sampling layer fuses the data from adjacent reference perspectives with the information of the current perspective. Finally, a decoder generates the inpainting output for the multi-frame degraded image. Experimental validation was conducted using an open dataset collected from ETH Zurich's binocular mobile platform, with results demonstrating that the proposed method outperforms commonly used algorithms, achieving superior inpainting effects in both qualitative visualization and quantitative evaluation metrics.