Leveraging sequence-to-sequence models for semantic annotation of Dutch pathology reports

Palga Foundation is responsible for indexing Dutch pathology data across the Netherlands, which relies on annotations of pathology reports. These annotations, derived from the conclusion text, consist of codes from the Palga thesaurus, serving patient care and scientific research. However, manual annotation by pathologists is both labor-intensive and prone to errors. Therefore, in this study, we seek to leverage sequence-to-sequence transformer models, particularly Text-To-Text Transfer Transformer (T5)-based models, to generate these annotations. Additionally, we investigate a constrained decoding (CD) approach that encodes domain knowledge. We compare a standard multilingual T5 model (mT5) with our own T5 model (PaTh5.NL) pre-trained using Palga data with the goal of better aligning the model's learned representations with the specific structure, terminology, and annotation conventions used in Dutch pathology reports. We fine-tune both pre-trained models using default (DD) and CD and compare both decoding strategies. Performance is assessed using Bilingual Evaluation Understudy (BLEU) scores for quantitative evaluation and case-based evaluations for qualitative assessment, where we use the generated codes to retrieve patients from the Palga database. Quantitative evaluations indicated that our two fine-tuned PaTh5.NL models significantly outperformed the fine-tuned mT5 model, particularly for shorter histology and cytology reports, but performance of all models declined on longer or complex reports. The case-based evaluation revealed that, despite higher BLEU scores, the PaTh5.NL models did not consistently outperform the mT5 model in retrieving relevant patients. This study demonstrates that fine-tuned T5-based models can enhance the annotation process for Dutch pathology reports, though challenges remain regarding complex conclusion texts, especially in histology and autopsy reports. Future research should focus on expanding gold-standard datasets and developing post-processing algorithms to improve annotations' generalization.