Learned Optical Flow-Based Defect Detection With Coarse-to-Fine Nonrigid Image Alignment

Detecting defects in printed circuit boards (PCBs) is essential to guarantee their reliability and functionality in industrial applications. Recently, automatic optical inspection (AOI) systems equipped with machine vision technologies have been increasingly used to identify defects in PCBs. Nevertheless, images captured by AOI systems frequently exhibit nonrigid deformations due to movement and vibrations, posing challenges to current techniques. Additionally, single-image-based defect detection methods often struggle to accurately identify ambiguous defects without the benefit of a reference template image. In this article, we present a novel reference-based coarse-to-fine image alignment framework with learned optical flow for PCB defect detection to address the above issues. Our approach begins with a spatial transformer network designed to estimate a global transformation matrix, effectively aligning the target and reference template images by exploring edge distance information. To handle local nonrigid deformations, we incorporate an optical flow estimation module with a multiscale diffeomorphism technique for estimating a more precise nonrigid optical flow field. Finally, we employ the Mahalanobis distance to identify defect locations, considering both the direction and magnitude of the estimated optical flow for each pixel. The generalization capability of the framework is improved due to the inherent generalization properties of optical flow. Extensive experiments on both the DeepPCB dataset and our real-world PCB dataset highlight the effectiveness of our framework, achieving commendable <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$F1$ </tex-math></inline-formula>-scores. These results confirm the robustness and versatility of our framework in accurately detecting a diverse range of defects commonly found in real-world scenarios. Our code is publicly available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/mxz520mxz/pcb-defect.git</uri>.