Plug-and-play illumination correction for illumination variant optical flow estimation

Optical flow estimation is widely used in various fields, such as motion scene understanding, autonomous driving, and object tracking. Despite great advances over the last decade, handling illumination variations in optical flow remains an open problem. Conventional optical flow estimation techniques build upon the brightness constancy assumption. Thus, variation in the lighting within the scene can affect the accuracy of optical flow algorithms. To tackle this challenge, here we report a plug-and-play illumination correction technique for robust optical flow estimation under variant illumination scenarios. This technique includes a motion-illumination decoupling strategy for the two images used to compute the optical flow, and an image brightness correction strategy. We trained a UNET-based neural network with Swin Transformer layer as basic block to decouple the motion and luminance information of the two images. Then we applied the decoupled luminance information from the reference image to the source image and adjusted its brightness. This technique makes both images maintain the same luminance for robust optical flow estimation under variant illumination conditions. We applied the reported technique on both traditional optical flow algorithms and deep learning-based optical flow algorithms, and the experiment results validated that it enables to enhance the algorithms’ illumination robustness, and achieves competitive evaluation results on the MPI-Sintel dataset and real-captured data.