Integrating Optical Flow into Deep Learning based Distortion Correction

One crucial task in 3D computer vision is the correction of geometric distortions, since most algorithms rely on the assumption that the image formation process can be described by a specific camera model, e.g. the pinhole camera model. In an autonomous driving scenario, however, the front-facing camera is most commonly placed behind the windshield, causing complex, nonlinear distortions. Previous attempts have been made to undistort such images using deep learning based methods. The input of these deep networks usually consists of one or more images, and they optionally include additional tasks such as semantic segmentation to improve the results. We hypothesize that the well-constrained nature of optical flow in rigid, static scenes provides useful cues for the process of image undistortion. By using optical flow as an additional input, we present a multi-view distortion correction method achieving superior results on both synthetic and real-world images compared to previous works, demonstrating the usability of optical flow for correcting highly complex distortions.