Accurate localization of characters in handwritten words

Abstract Providing precise character-level feedback on handwritten text requires systems capable of localizing individual characters. However, existing handwriting datasets typically lack character-level annotations, limiting the development of localization models. In this work, we introduce the first large-scale English dataset (called CharLoc) with detailed bounding boxes and labels for over 219,000 handwritten characters across 37,000 word images from 40 writers. Using the CharLoc dataset, we train and evaluate bounding-box and contour models for character localization. We show that using the manually annotated data substantially improves character localization over conventional baselines and enables two state-of-the-art systems to achieve substantially higher localization performance than when trained on synthetic data. At the same time, models trained on CharLoc can not only localize characters but also recognize them, achieving performance only slightly below state-of-the-art systems that lack localization capabilities. We demonstrate the robustness of a combination of both capabilities in an out-of-distribution case study with newly collected samples, where we are able to accurately highlight wrong characters in misspelled words, paving the way to personalized feedback in handwritten texts.