Word Image Representation at Local and Global Levels Based on Vision Transformers

Recognizing handwritten words poses a challenge due to the wide variability in writing styles and the potential blurriness and noise present in document images. This study presents an end-to-end neural network that integrates inception modules and transformers to efficiently encode a nd recognize noisy word images. The contributions of this work are: 1- A robust word image representation achieved by utilizing CNN Inception modules to capture local features, Transformer encoders to model global dependencies, and bidirectional GRU layers to enhance sequence processing. 2- An efficient lexicon correction mechanism incorporating an LRU (Least Recently Used) cache to ensure efficient performance even with large lexicons. Additionally, we discuss a probabilistic approach for recognizing both in-vocabulary and out-of-vocabulary words, solved at the same time as a sequence-to-sequence recognition and a similarity matching, which allows implicitly to correct the predictions. The proposed model shows promising performance compared to state-of-the-art methods on the IAM and RIMES datasets in terms of Word Error Rate (WER) and Character Error Rate (CER), highlighting the robustness and adaptability of the proposed model.