MSKE-LLM: Multi-Stage Knowledge Enhancement Policy Question-Answering Large Language Model

We present MSKE-LLM, a Multi-Stage Knowledge Enhancement (MSKE) policy question-answering Large Language Model (LLM). Its goal is to use structured policy knowledge graphs to enhance knowledge base retrieval ability, thereby improving the knowledge matching accuracy and response generation performance of LLMs when doing policy question-answering, a task to provide answers to questions regarding policies and regulations according to the relevant knowledge. Specifically, in response to the general inadequacy of knowledge in LLMs within the policy domain, and the decrease in retrieval accuracy as knowledge base size increases, we gather policy documents from local policy websites. Subsequently, we build an expansive policy knowledge graph to facilitate high-precision policy knowledge base matching, elevating knowledge retrieval accuracy and answer quality. In addition, we propose a standard for evaluating multi-stage knowledge enhancement policy question-answering LLM and conduct multi-dimensional human and automated evaluation and quantitative ablation research. The experimental results show that the proposed MSKE-LLM is superior to the existing models.