Plug-and-play algorithm for 3D guided video super-resolution of single-photon LiDAR data

Single-photon avalanche diodes (SPADs) are advanced sensors capable of detecting individual photons and recording their arrival times with picosecond resolution using time-correlated single-photon counting (TCSPC) detection techniques. They are used in various applications, such as LiDAR and low-light imaging. These single-photon cameras can capture high-speed sequences of binary single-photon images, offering great potential for reconstructing 3D environments with high motion dynamics. To complement single-photon data, these cameras are often paired with conventional passive cameras, which capture high-resolution intensity images at a lower frame rate. However, 3D reconstruction from SPAD data faces challenges. Aggregating multiple binary measurements improves precision and reduces noise but can cause motion blur in dynamic scenes. Additionally, SPAD arrays often have lower resolution than passive cameras. To address these issues, we propose a novel computational imaging algorithm to improve the 3D reconstruction of moving scenes from SPAD data by addressing the motion blur and increasing the native spatial resolution. The goal is to turn the high-speed SPAD events, recorded at a high frame rate, into non-blurred high-resolution depth images at the frame rate of the passive sensor. We adopt a plug-and-play approach within an optimization scheme alternating between guided video super-resolution of the 3D scene, and precise image realignment using optical flow. Experiments on synthetic data show that our method significantly improves image resolution across various signal-to-noise ratios and photon levels. We validate our method using real-world SPAD measurements in three practical situations with dynamic objects. First on fast-moving scenes (i.e. fan) in laboratory conditions at a short range (3 meters); second very low-resolution imaging of people with a consumer-grade SPAD sensor from STMicroelectronics; and finally, high-resolution imaging of people walking outdoors in daylight at a range of 325 meters under eye-safe illumination conditions using a short-wave infrared SPAD camera. These results demonstrate the robustness and versatility of our approach.