Amharic medical question answering system using AmhBERT

A medical question-answering (QA) system is a natural language processing (NLP) application designed to answer medical questions. Developing such systems for Amharic remains challenging because of the limited availability of medical language resources. To address this gap, this study presents Amharic medical question answering system based on Amharic adapted multilingual BERT. First, we prepared medical question answering dataset from freely accessible medical sources such as Ethiopian Health Institute (EHI). Following data collection, we extracted relevant context passages from the documents and used them to generate question-answer pairs. This process produced a collection of context-question-answer triples, each consisting of a health-related context, a question, and answer. The quality of the medical question–context–answer triples was evaluated by three medical experts from Tibebe Ghion Hospital, Bahir Dar University, with expertise in clinical nursing, general medical practice, and maternal and child health. The annotators use two criteria: question–context support and answer correctness. Pairwise Cohen’s kappa was used to assess inter-annotator agreement, with a mean score of 0.872 indicating a high level of agreement for question–context support. The dataset was evaluated using Amharic adapted multilingual BERT (mBERT) and four other transformer models. AmhBERT was obtained by continually pretraining multilingual BERT (mBERT) on 1.34 + million Amharic text sentence. Experimental results show that AmhBERT achieved the best performance over XLM-R, mBERT, BERT, and RoBERTa by substantial margins.