Multimodal Handwritten Exam Text Recognition Based on Deep Learning

To address the complex challenge of recognizing mixed handwritten text in practical scenarios such as examination papers and to overcome the limitations of existing methods that typically focus on a single category, this paper proposes MHTR, a Multimodal Handwritten Text Adaptive Recognition algorithm. The framework comprises two key components, a Handwritten Character Classification Module and a Handwritten Text Adaptive Recognition Module, which work in conjunction. The classification module performs fine-grained analysis of the input image, identifying different types of handwritten content such as Chinese characters, digits, and mathematical formula. Based on these results, the recognition module dynamically selects specialized sub-networks tailored to each category, thereby enhancing recognition accuracy. To further reduce errors caused by similar character shapes and diverse handwriting styles, a Context-aware Recognition Optimization Module is introduced. This module captures local semantic and structural information, improving the model’s understanding of character sequences and boosting recognition performance. Recognizing the limitations of existing public handwriting datasets, particularly their lack of diversity in character categories and writing styles, this study constructs a heterogeneous, integrated handwritten text dataset. The dataset combines samples from multiple sources, including Chinese characters, numerals, and mathematical symbols, and features high structural complexity and stylistic variation to better reflect real-world application needs. Experimental results show that MHTR achieves a recognition accuracy of 86.63% on the constructed dataset, significantly outperforming existing methods. Furthermore, the context-aware optimization module demonstrates strong adaptive correction capabilities in various misrecognition scenarios, confirming the effectiveness and practicality of the proposed approach for complex, multi-category handwritten text recognition tasks.