Research on Depth Image Super-resolution Algorithm for High and Low Frequency Feature Modulation Fusion Guided by Color Images

Depth images effectively describe the information of a 3D scene.However,the acquisition equipment and imaging environment limit the resolution and high-frequency information of the depth images acquired by depth sensors.It is imperative to improve the resolution of depth images.Some depth map super-resolution algorithms have significantly improved their performance by introducing RGB images from the same scene to provide guidance information for the depth map super-resolution process.The key challenge lies in effectively leveraging the RGB information to guide the depth map super-resolution reconstruction process,addressing the modal inconsistency between the depth map and RGB images.Existing methods primarily focus on high-frequency information,overlooking the low-frequency global information crucial for algorithm performance.To address these limitations,this paper proposes a novel color image-guided,high and low-frequency feature modulation fusion super-resolution reconstruction algorithm for depth maps.A two-branch feature extraction module extracts high and low frequency features from color and depth images,respectively.CNN and Transformer are used in each branch to extract local high frequency and global low frequency information.A two-way transformation and fusion between high frequency information and low frequency information of color and depth images is achieved by constructing a two-way modulation module.The model fully exploits the complementary information between the depth image and the color image.It uses a bidirectional modulation within different modes and different frequencies and the subsequent fusion of high and low-frequency information.The depth super-resolution algorithm based on the guidance of the color image can achieve better reconstruction results.The lossless information compression using reversible neural network INN extracts high-frequency detail information more effectively,and the quadtree attention mechanism reduces the computational complexity of the Transformer in extracting global information,improving the efficiency of the algorithm.The experimental results on the public datasets show that the proposed method outperforms the comparison methods in both quantitative and qualitative aspects,achieving better subjective visualization results.