$$\hbox {D}^2$$RSM-former: a directional deformable refinement transformer with soft-edge smooth span mask for robust handwritten text recognition

Abstract Handwritten text recognition technology plays a crucial role in historical document digitization, archive management systems, and cultural heritage preservation. Although existing methods based on Convolutional Neural Networks and Vision Transformers have achieved remarkable progress in feature extraction and global modeling, they still exhibit limitations when addressing inherent challenges of handwritten text, including stroke deformation, character adhesion, and cross-lingual robustness. To address these challenges, we propose a Directional Deformable Refinement Transformer with Soft-edge Smooth Span Mask for Robust Handwritten Text Recognition ( $$\hbox {D}^2$$ D 2 RSM-Former), which innovatively integrates a Hierarchical Adaptive Directional Deformable Feature Extractor (HADD-FE), a Soft-edge Smooth Span Mask (SSSM) strategy, and a Heterogeneous Interactive Refinement Encoder (HIREncoder). HADD-FE dynamically adjusts receptive field configurations through an adaptive directional deformable mechanism, effectively capturing anisotropic features of handwritten text. The SSSM enhances generalization through smooth boundary transitions and adaptive length sampling. HIREncoder achieves synergistic modeling of local–global features through dialogic head contextual attention and heterogeneous iterative refinement mechanisms. Extensive experiments on six benchmark datasets demonstrate that, without relying on any pre-training or synthetic data, $$\hbox {D}^2$$ D 2 RSM-Former achieves state-of-the-art performance on READ2016, LAM, and IAM datasets, while also demonstrating highly competitive results on RIMES2011, BRESSAY, and VOHTR2018 datasets.