RARoK:Retrieval-Augmented Reasoning on Knowledge for Medical Question Answering

Although large language models (LLMs) perform impressively in natural language tasks, they face several challenges, such as conventional new knowledge, generating accurate responses, and explaining their reasoning. To address these issues, we propose a new approach, Retrieval-Augmented Reasoning on Knowledge (RARoK), which combines Chain of Thought (CoT) prompts with Retrieval-Augmented Generation (RAG). By leveraging external information from knowledge graphs (KGs), RARoK iteratively refines the CoT to further optimize the reasoning process of the model. Our approach significantly outperformed the baseline in various Q&A tasks, especially in the medical domain. Compared to the SOTA method, RARoK improves Hits@1 by 3.3% on the WebQSP dataset, while improving the Hits@1 and F1 scores by 18.6% and 10.4% respectively on the CWQ dataset. The experimental results on the Knowledge Graph Question Answering (KGQA) datasets and the Medical Q&A datasets show that our method has better reasoning ability and interpretability compared to the vanilla LLMs and other retrieval-enhanced methods. Our code and data are publicly available (https://github.com/cskyan/RARoK).