From Policy Documents to Audit Logic: A LLMs-based Framework for Extracting Executable Audit Rules

Generative Artificial Intelligence (GenAI), particularly Large Language Models (LLMs), has shown great potential across diverse fields. This study explores leveraging LLMs to efficiently parse complex medical policy documents and convert them into executable audit rules. Although focused on hospital auditing—a domain characterized by dense policies and well-defined audit logic—our approach is broadly applicable to public administration. First, we formally model hospital audit rules using ProLog syntax. Second, we propose a column compression method based on text chunking to reduce input size and improve LLMs inference efficiency. Third, we develop prompt-based extraction combined with semantic alignment to enhance accuracy. Experimental results demonstrate that our method not only reproduces rules consistent with expert-crafted knowledge but also uncovers numerous novel, unexpected, yet valid audit rules. Compared to generic information extraction approaches, our framework yields significantly better performance. This work advances policy document information extraction and offers significant practical value for expanding hospital audit coverage.