Efficient Depth and RGB Camera Fusion Algorithm for Depth Imaging with Improved Spatial Resolution

Cameras for capturing depth images typically suffer from low spatial resolution, whereas RGB cameras offer high resolution. A common approach to address these drawbacks of depth cameras is to combine them with RGB cameras using a fusion algorithm. We propose, implement and benchmark a new spatial upscaling algorithm based on edge extrapolation and compare it to state-of-the-art spatial upscaling methods. The edge extrapolation algorithm is very efficient and achieves cutting edge performance. It produces high-resolution depth maps with particularly clean edges, but like other state-of-the-art methods, it also exhibits certain specific artifacts. The new algorithm first interpolates the depth image and combines it with an RGB image to extract an edge map. The blurred edges of the interpolated depth map are then replaced by an extrapolation from neighboring pixels for the final high-resolution depth map. The algorithm is benchmarked using the Middlebury and DIML datasets, as well as a number of new 3D time-of-flight depth images.