Handwritten English Recognition Based on CNN and CNN- Head Transformer

Traditional handwritten text recognition methods typically rely on Convolutional Neural Networks (CNN) for image classification tasks. However, these methods inadequately capture semantic features, leading to limitations when dealing with long sequence data. To address this issue, this paper proposes a method that combines CNN and Transformer models for handwritten English text recognition. Initially, a CNN is utilized to extract features from the input images, and these feature sequences are then fed into a Transformer encoder with a CNN head. The encoder integrates the functionality of CNNs across multiple attention heads, keeping 2 attention heads unchanged while obtaining value vectors from 6 attention heads of the Transformer. This proposed method effectively captures global semantic information, enhancing the model’s ability to extract local features and potential word information. Additionally, adaptive gated residual connections are applied to merge features from the current layer and sub-layers, improving the model’s capability to capture both local and global features of handwritten English text. A Connectionist Temporal Classification (CTC) decoder is used to decode the features output by the encoder, producing the final recognition results. Experimental simulations using the TAL_OCR_ENG dataset demonstrate that this method effectively recognizes sentences, achieving a recognition accuracy of $\mathbf{9 4. 3 3 \%}$, a character error rate (CER) of 3.31%, and a word error rate (WER) of $\mathbf{1 2. 5 2 \%}$. Compared to traditional methods, this approach shows a significant improvement in accuracy, indicating higher performance in handwritten English text recognition.