Handwritten German Administrative Forms (HGAF): A Dataset for Handwriting Recognition with Fine-Tuned and Zero-Shot Baselines

Abstract Administrative processes remain largely paper-based,and handwritten form contents must be transferred into structured data. Whereforms carry personal information, the proprietary cloud services that leadpublished benchmarks are not suitable, so only on-premise models remain. Currently, nopublic corpus supports this setting: handwriting datasets hold continuous prose,and form datasets hold printed text. Therefore, a dataset of 291 German administrative forms, named HandwrittenGerman Administrative Forms (HGAF), was collected from 95 writers. Each writercompleted three layouts, A, B and C, with identical fields and identicalsynthetic contents. The visual structure therefore varies, while writer andcontent stay constant. The corpus carries no personal information and can bereleased. Two approaches were implemented and scored per field: a specialistpipeline around a fine-tuned line-level TrOCR model, and three open-weightvision-language models (VLM) applied without fine-tuning. On layout A, the only layout used for fine-tuning, twozero-shot models reach 4.5% and 4.7% character error rate (CER), against6.8% for the specialist pipeline. On layouts B and C, the gap widens.No on-premise configuration transcribed more than 7.7% of forms without anerror. In this scenario, VLMs reached the best performance. Although they do not requirefine-tuning, they may still require configurationadjustments to fulfil the HTR task properly. We recommend prompting a generalinstruction-following model with a fixed output schema. This removes theextraction stage and allows the system to be adapted to different form layoutswith little effort.