Adapting Large Language Models for Biomedicine though Retrieval-Augmented Generation with Documents Scoring

By integrating the generative capabilities of Large Language Models (LLMs) with external biomedical knowledge repositories, Retrieval-Augmented Generation (RAG) offers significant potential for addressing knowledge-intensive biomedical tasks, enhancing the precision and effectiveness of clinical decision-making. However, the standard RAG approach fails to fully leverage the information within the retrieved documents, which can lead to incorrect response generation. While advanced RAG approaches empower LLMs to understand the correlation between queries and documents, they require costly annotation. In this work, we propose a strategy to help LLMs better leverage the information within retrieved documents in RAG system while reducing data acquisition costs. We fine-tuned an LLM on a dataset comprising queries, retrieved documents, and their relevance scores generated by a pre-trained biomedical re-ranker to adapt the model for answering questions from reference documents. The fine-tuned model can independently score retrieved documents before answering the question, thereby more effectively utilizing the information contained within retrieved documents. Experimental results shows that we have improved the model’s accuracy on two biomedical benchmarks while reduced the costs of data acquisition.