Multilingual Party model for European languages

Party for European Languages Party is page-wise recognition of text-y. It is a replacement for conventional text recognizers in automatic text recognition pipelines that utilize either bounding box or baseline+bounding polygon segmentation methods for layout analysis. This is a model for the recognition of print and handwriting in a number of European languages using the most recent party release with language token support: Ancient Greek Catalan Church Slavonic Corsican Czech Dutch English Finnish French German Irish Latin Lithuanian Middle Dutch Middle French Norwegian Occitan Picard Polish Portuguese Romanian Russian Serbian Slovenian Spanish Ukrainian Serbian has been trained only on Cyrillic script. Architecture The recognizer is a deep fusion multimodal model consisting of a Swin vision encoder and a tiny Llama (100M parameters) decoder trained with octet tokenization. The network is prompted with the line positions through positional embeddings added to the encoder hidden state. During training the encoder weights were initialized with a ImageNet-22k pretrained Swin-base from pytorch-image-models, the decoder weights came from a custom Llama 3.2 pretrained on a subset of OSCAR 2301 tokenized with a ByT5-style octet tokenizer. The pre-initialized model was then pre-trained on a collection of public and private training historical document page datasets augmented with born-digital data crafted from PubLayNet. Uses This model is a recognition foundation model primarily targeted at automatic text recognition for the humanities. While it produces fairly accurate output on an impressive range of material it is intended to be fine-tuned on some target dataset to ensure compliance with desired transcription guidelines. Transcription guidelines, Normalization, and Transformations No attempts have been made to normalize the datasets or to only use data adhering to common transcription guidelines. While some subsets of the corpus are internally consistent, only a very small subset of the languages in the training data only contain datasets from a single source. Bias, Risks, and Limitations The training corpus frequently incorporates datasets of esoteric material transcribed for specific purposes. Especially machine-printed and born-digital material lack diversity, so error rates will most likely vary considerably across languages and document type. Some additional limitations are to be expected: Some transcriptions resolved abbreviations while others did not. Inconsistent output is to be expected, in particular for European manuscripts in Latin script. As the model predicts 8-bit UTF-8 code units directly the lack of consistent Unicode normalization can cause slightly different code point streams during prediction. How to Get Started with the Model Install the party package from github and follow the instructions. Training Details Training Data This model has been fine-tuned from a very generic base model with datasets containing writing from European languages, principally in Latin script but also Cyrillic, Greek, and Glagolithic. |Language|Pages|Lines|Datasets| |---|---|---|---| |Catalan| | |FONDUE-CA-PRINT-20| |Corsican| | |HN2021-OCR-Poesie-Corse| |Czech| | |Padeřov-Bible-handwriting-ground-truth| |Dutch| | |ATR_TrainingSet_NLF_Newseye_GT_SV_M2+4 private manuscript datasetsVOC dataset| |English| | |FONDUE-EN-PRINT-20PubLayNetUniversity of Denver CollectionsJoseph Hooker HTRCCCC MS 41| |Finnish| | |NewsEye/READ OCR Finnish Newspapers| |French| | |NewsEye READ AS French NewspapersBoccaceFabliauxLiberCremma MedievalDecameronFRFONDUE-FR-MSS-18FONDUE-FR-MSS-19FONDUE-FR-PRINT-16FONDUE-FR-PRINT-17FONDUE-FR-PRINT-20Données imprimés gothiques du 16e siècleDonnées HTR incunables du 15e siècleDonnées HTR manuscrits du 15e siècle"Tables Décennales" French Civil RegistryDonnées imprimés du 16e siècleDonnées imprimés du 17e siècleDonnées imprimés du 18e siècleIncunable français du 15e siècleHTRomanceHTR-SETAF-Jean-MichelHTR-SETAF-LesFaictzJCHHTR-SETAF-Pierre-de-VingleLa Correspondance Jacques Doucet - René JeanOCR17+Tapus CorpusTIMEUS CorpusRecensement Valaisan3 private handwritten and print datasets| |German| | |Charlottenburger AmtsschrifttumDACH GTDigiTue GTFibelnFONDUE-DE-MSS-18FoNDUE_Wolfflin_FotosammlungHKB GTGround truth for Neue Zürcher Zeitung black letterReichsanzeiger GTStABS Ratsbücher O10NewsEye / READ OCR Austrian NewspapersWeisthuemer3 private manuscript datasets| |Greek| | |EPARCHOSHTR CPgr23Handwritten Paleographic Greek Text RecognitionΧΦ114XΦ79ΧΦ5310 small private manuscript datasets| |Italian| | |episearch-htrFONDUE-IT-PRINT-20HTRomance Italian1 private print dataset| |Latin| | |Caroline MinusculeCREMMA-Medieval-LATHTRomance LatinDIVA-HisDBEutychesFONDUE-LA-MSS-MAFONDUE-LA-PRINT-16Lateinische GedichteWien ÖNB Cod 21602 private manuscript datasets| |Multilingual| | |FONDUE-MLT-ART[FONDUE-MLT-CAT](https://github.com/FoNDUE-HTR/FONDUE-MLT-CAT)[FONDUE-MLT-PRINT-TEST](https://github.com/FoNDUE-HTR/FONDUE-MLT-PRINT-TEST)gt_structure_text| |Portuguese| | |Portuguese Handwriting 16th-19th c.| |Russian| | | |1 private manuscript dataset| |Spanish| | |FONDUE-ES-PRINT-19FoNDUE-Spanish-chapbooks-DatasetHTR AraucaniaHTRomance Spa3 private manuscript datasets| For ancient Greek, Czech, Dutch, Finnish, Irish, Latvian, Lithuanian, Polish, Romanian, Russian, Serbian, and Slovenian additional synthetic print data generated with the pangoline tool was used. Training Procedure and Hyperparameters Training regime:: 6 * A40 GPU, BF16-mixed precision, Mars-AdamW optimizer with caution, batch size: 18, gradient accumulation: 8, effective batch size: 864, 12 epochs with 5000 iteration warmup and cosine decay, max LR 1e-4, min LR 1e-6 at end of epoch 12, weight decay 1e-5, gradient clipping 1.0, augmentation, random sampling of bbox and curve batches Evaluation Testing Data, Factors & Metrics Testing Data {{ testing_data | default("[More Information Needed]", true)}} Factors {{ testing_factors | default("[More Information Needed]", true)}} Metrics CER: WER: Summary {{ results_summary | default("", true) }} Citation [optional] BibTeX: {{ citation_bibtex | default("[More Information Needed]", true)}}