License Plate Recognition Using Three-Dimensional Rotated Character Recognition and Instance Segmentation by Deep Learning

License plate recognition is currently used in various situations, such as parking lot vehicle management and the tracking of wanted vehicles. These applications require recognition from all possible camera angles. Therefore, we propose a method for license plate recognition in images captured from various camera angles. First, the license plate area is detected from the input image using the YOLOv5 object detection method. The subsequent process can be performed in two manners: by rotating the license plate to the front using projective transformation and trapezoidal correction, or by using graph matching. The first method performs Hough transforms or linear approximations as preprocessing, and then calculates the rectangle surrounding the license plate from the YOLOv5 output. After performing trapezoidal correction, the maximally stable extremal regions (MSER) are used to detect character candidates, and characters are recognized using 3D rotated character recognition. The second method performs character candidate detection and recognition without trapezoidal correction, followed by graph matching. These methods are versatile because they do not depend on the layout of license plates, which varies among countries and regions. Two types of datasets were used: a set of images containing Japanese license plates collected by ourselves, and a set of publicly available images containing Taiwanese license plates. Two recognition rates were evaluated: the license plate character recognition rate and license plate recognition rate in which all characters in one plate are recognized. The license plate character recognition rates using graph matching were 91.9% for the Japanese license plates and 89.3% for the Taiwanese license plates. The license plate recognition rates were 84.0% and 84.5%, respectively.