DKITNLP at ArchEHR-QA 2025: A Retrieval Augmented LLM Pipeline for Evidence-Based Patient Question Answering

This paper describes our submission for the BioNLP ACL 2025 Shared task on grounded Question Answering (QA) from Electronic Health Records (EHRs).The task aims to automatically generate answers to patients' health related questions that are grounded in the evidence from their clinical notes.We propose a two stage retrieval pipeline to identify relevant sentences to guide response generation by a Large Language Model (LLM).Specifically, our approach uses a BioBERT based bi-encoder for initial retrieval, followed by a reranking step using a fine-tuned cross-encoder to enhance retrieval precision.The final set of selected sentences serve as an input to Mistral 7B model which generates answers through few-shot prompting.Our approach achieves an overall score of 31.6 on the test set, outperforming a substantially larger baseline model LLaMA 3.3 70B (30.7), which demonstrates the effectiveness of retrieval-augmented generation for grounded QA.