Toward a Large Language Model-Driven Medical Knowledge Retrieval and QA System: Framework Design and Evaluation

Recent advancements in large language models (LLMs) have driven remarkable progress in text processing, opening new avenues for medical knowledge discovery. In this study, we present ERQA, a mEdical knowledge Retrieval and Question-Answering framework powered by an enhanced LLM that integrates a semantic vector database and a curated literature repository. The ERQA framework leverages domain-specific incremental pretraining and conducts supervised fine-tuning on medical literature, enabling retrieval and question-answering (QA) tasks to be completed with high precision. Performance evaluations implemented on the coronavirus disease 2019 (COVID-19) and TripClick datasets demonstrate the robust capabilities of ERQA across multiple tasks. On the COVID-19 dataset, ERQA-13B achieves state-of-the-art retrieval metrics, with normalized discounted cumulative gain at top 10 (NDCG@10) 0.297, recall values at top 10 (Recall@10) 0.347, and mean reciprocal rank (MRR) = 0.370; it also attains strong abstract summarization performance, with a recall-oriented understudy for gisting evaluation (ROUGE)-1 score of 0.434, and QA performance, with a bilingual evaluation understudy (BLEU)-1 score of 7.851. The comparable performance achieved on the TripClick dataset further underscores the adaptability of ERQA across diverse medical topics. These findings suggest that ERQA represents a significant step toward efficient biomedical knowledge retrieval and QA.