Ranking handwriting legibility with foundation-model embeddings at human-level consistency on the HaLO dataset

Abstract Legibility reflects the visual quality of handwriting and can signal difficulties in underlying motor and cognitive processes as well as developmental disorders. Early identification of handwriting problems is therefore crucial for providing targeted support. However, automated legibility assessment remains underexplored, partly due to the subjective nature of legibility and limited annotated data. To address this, we introduce HaLO, a dataset of 1, 907 handwriting images from 202 students, annotated with 19, 070 pairwise legibility judgments, and release its image features and annotations. We propose a comparative ranking approach for legibility assessment that leverages relative judgments and global rankings instead of absolute scores. Using a linear probe on Pixtral 12B embeddings, we identify which of two samples is more legible with strong performance ( $$\textrm{ACC}=0.83$$ , $$\mathrm {ROC\,AUC}=0.92$$ ), approaching the ceiling implied by the inconsistency of the annotations. The induced ranking reaches human-level consistency ( $$\tau =0.70$$ , $$\rho =0.89$$ ), and the inferred ordinal scores match or exceed inter-annotator agreement ( $$\alpha \ge 0.61$$ ). Additional concept-vector analyses suggest that pretrained Pixtral-ViT embeddings linearly encode interpretable visual properties related to legibility, including cursiveness, stroke thinness, and image aspect ratio. Overall, this work introduces a novel dataset and a ranking approach for automatic handwriting legibility assessment, demonstrating that a linear probe on foundation-model image representations can achieve human-level performance.