Manual annotation of Robson criteria and obstetric entities: Inter-annotator agreement and initial NER models implementation

We present a manually annotated dataset focused on Robson Criteria Classification, encompassing entities related to obstetric variables, antibiotics, uterotonics, complications, delivery outcomes, and personal information. A total of 1,865 Electronic Health Records (EHRs) were annotated, yielding 18,105 labeled entities. Inter-annotator agreement (IAA) was assessed using F1 and Kappa scores. The F1 and Kappa scores were 0.86 and 0.89 for strict criteria, respectively. For relaxed criteria, F1 and Kappa scores were 0.93 and 0.94, respectively. Using the annotated dataset, we trained several Named Entity Recognition (NER) models: Conditional Random Fields (CRF), fine-tuned BERT, RoBERTa, and XLM-RoBERTa models, and a zero-shot and fine-tuned Llama 3.1-8B-Instruct model. Models were evaluated under two scenarios: a 23-label scheme and a 46-label IOB scheme. The CRF model achieved the highest precision, with scores of 0.69 and 0.74 for 23 and 46 labels, respectively. RoBERTa and XLM-RoBERTa models demonstrated the best recall and F1-scores, achieving 0.70 and 0.76 for recall on 23 and 46 labels, respectively, and 0.65 and 0.72 for F1-scores on 23 and 46 labels, respectively.