Improving OCR Through Transfer Learning with HTR Models : A Transfer Learning Study with TrOCR

Background In the era of digital transformation, the ability to convert physical documents into machine- readable format has been crucial. OCR (Optical Character Reader) and HTR (Handwritten Text Recognition) are two technologies at the forefront of this revolution. While OCR has been widely used for recognizing printed text, HTR has been instrumental in deciphering handwritten text. Despite their individual successes, there remains a significant scope for improvement in their performance, and different approaches have been explored since OCR was discovered. As discussed in the paper [1], Shafique et al. highlights the challenges and diversity of approaches used in HTR tasks. One promising approach has been the integration of machine learning techniques. The integration of machine learning into OCR tools has revolutionized the efficiency and accuracy of text scanning and understanding. Using advanced algorithms, particularly deep learning models such as CNNs (convolutional neural networks), OCR systems can now recognize and differentiate between various fonts, styles, and even handwritten text with remarkable precision [2]. These tools have become context aware, enabling them to handle complex lay-outs and noisy images, there by organizing extracted text more effectively. Continuous learning capabilities allow OCR systems to improve with time, adapting to new types and layouts of text [3]. In addition, machine learning improves pattern recognition, making it easier to extract relevant information from documents with repetitive structures, such as invoices or forms. Real-time processing capabilities have also been significantly improved, providing instant results for applications such as mobile scanning apps. In general, the integration of machine learning has transformed OCR tools into sophisticated high-precision systems that streamline the process of scanning and understanding text, making it faster, more accurate and user-friendly. As seen in the paper [4], Gupta et al. details various machine learning algorithms applied in OCR and their effectiveness in enhancing the performance of text scanning and understanding. However, despite these advances, OCR systems are still considered to face significant challenges in accurately recognizing text. Factors such as insufficient datasets,ageing documents, poor image quality, variations in ink abundance, and complex page layouts [5]. These factors can distort the characters, making it difficult for the OCR system to correctly identify them. Furthermore, traditional OCR systems typically identify characters based on the small space between them, which can lead to errors if the spaces are not clearly defined or if the characters are connected. No tool to date can recognize all text with 100 percent accuracy, with GCP and AWS Textract coming in with an almost perfect 99.3 percent [6]. In the study, we can see that tools with an almost perfect score utilize an approach where they include handwritten text images during training. This thesis explores the viability of using transfer learning techniques to enhance the performance of OCR systems, particularly in situations with insufficient datasets. Transfer learning, a method where a pre-trained model on a large dataset is fine-tuned for a specific task, offers a promising solution to the challenges faced by OCR systems in recognizing text from under-represented languages. By leveraging models trained on general to extensive datasets, this study will investigate how transfer learning can be applied to improve the accuracy and efficiency of OCR tools in processing languages with limited data resources. Additionally, this thesis will explore the potential of integrating transfer learning techniques with HTR models. HTR models are adept at understanding complex characters and words, making them a valuable asset for OCR systems. This thesis looks at whether the complexity and abilities of HTR models can improve OCR performance. The goal is to find the best ways to use HTR models to improve text recognition in linguistic contexts. Objectives Transfer learning, a concept in machine learning, allows the knowledge gained from one task to be applied to another related task. In our context, the ‘knowledge’ is the ability of HTR models to handle variability and ambiguity in recognizing handwritten text. The hypothesis is that this ‘knowledge’ can be transferred to improve OCR’s performance on printed text, especially those that resemble handwritten text or have poor quality. The main objectives of this thesis will be to address the following topics: An investigation into how well an OCR model pre-trained on handwritten text performs OCR tasks when compared to an OCR model trained solely on printed text. This includes comparing their performance on a common dataset. A thorough analysis of how to effectively apply HTR principles to enhance OCR accuracy follows. The analysis includes transferring features, techniques, and knowledge from HTR to OCR. Since the thesis will focus on one under-represented language, in terms of existing research and developments available, the thesis will mainly focus on Swedish language. Eventually, this leads to the development of datasets that are needed to train and conduct comparative analysis. Hence, both synthetic printed and handwritten dataset generator tools are expected to be an objective for the thesis. The following chapters will touch on literature review, methodology, experiments conducted, and the results obtained. The overall objective is to provide a comprehensive understanding of how transfer learning can bridge the gap between HTR and OCR, thereby pushing the boundaries of what is currently achievable in the field of text recognition.