LLM-Driven Compliance Checking for Natural-Language Policies over Data Product Descriptors

Abstract Modern data architectures increasingly rely on decentralized data products described through semi-structured artifacts, such as YAML descriptors, to support scalability, interoperability, and governance. Ensuring that these descriptors comply with organizational and regulatory policies remains a challenging and labor-intensive task, largely due to the natural-language nature of policies and the heterogeneity of descriptor schemas. In this paper, we investigate a model-based approach to evaluating natural-language policies over semi-structured data product descriptors expressed in YAML, leveraging Large Language Models (LLMs) as reasoning components. We formalize the policy-descriptor compliance task as a binary decision problem and propose an LLM-based microservice architecture that combines structured prompt engineering with deterministic output constraints to support automated policy verification. To evaluate the approach, we construct an augmentation-based benchmark comprising 4,000 policy-descriptor pairs derived from 40 manually annotated seed combinations. The benchmark is designed to assess model robustness to policy paraphrases and semantically equivalent descriptor rewrites. We evaluate four open-source instruction-tuned LLMs under identical conditions and report results in terms of accuracy, precision, recall, F1-score, and execution time. Experimental results show that LLMs can effectively support policy verification in semi-structured environments, achieving up to 72% accuracy. However, we observe a consistent conservative bias toward non-compliance, as well as systematic failure modes involving conjunctions, bidirectional constraints, and conditionally inapplicable rules. Our analysis highlights both the potential and the current limitations of LLM-driven governance, and suggests that expressing policies as explicit procedural checks substantially improves validation reliability.