The Evaluation of Transformer Models for the Detection of Adverse Drug Events: A Benchmark Study Using Dutch Free-Text Documents of Hospitalized Patients

The Dutch domain-specific MedRoBERTa.nl showed the best performance in detecting ADEs in Dutch clinical texts, and in line with previous studies in English language settings, outperformed Bi-LSTM. The inclusion of external validation highlights its generalization potential. Our findings also underline the need for further model improvement and use of performance measures suited to rare outcomes such as ADEs, as micro-averaged F1 scores inflate performance compared with macro-averaged F1 scores. We provide a robust and clinically meaningful benchmark approach for NLP-based ADE detection in clinical free-text documents. Our approach can serve as a guidance for future NLP benchmarks in ADE domain.