Policy-As-Code Governance For Bronze-Layer Ingestion In Regulated Databricks Medallion Lakehouse Architectures
Modern data platforms widely use the medallion architecture to organize pipelines on lakehouse systems such as Databricks. Governance controls, including schema validation, sensitive-data handling, retention, lineage tracking, quality checks, and access management, are typically applied in downstream layers or through manual processes. This leaves the bronze layer, where raw data first enters the platform, inconsistently governed. In regulated sectors, a lapse at this layer, such as an unhandled schema change or unmasked sensitive field, can spread across the pipeline and create compliance, financial, and operational risk, because downstream analytics and machine learning depend on the integrity of this raw foundation. This paper reviews and synthesizes existing governance, data quality, and policy-as-code approaches, then proposes a unifying framework that expresses governance requirements as declarative, version-controlled policies enforced automatically at bronze ingestion. The framework combines a policy repository, an enforcement engine with allow, warn, quarantine, and reject outcomes, and an append-only audit store, using Databricks-native tools and general policy engines. The synthesis shows that six governance concerns, namely schema, sensitive data, retention, lineage, quality, and access, can be coordinated by a single policy layer in which each concern is handled by the mechanism suited to it: an ingestion-time gate for schema, sensitive data, and quality; a lifecycle policy for retention; metadata capture for lineage; and access control applied at query and access time. The contribution is a unified policy layer that coordinates these six concerns with continuous auditing, rather than forcing all of them through one enforcement point. Novelty is claimed relative to the approaches examined in this review.
Authors
- Shikha Sharma
Publication Details
- Journal
- International Journal of Advanced Artificial Intelligence Research
- Published
- 2026-10-27
- DOI
- https://doi.org/10.55640/ijaair-v03i10-03
- Primary Topic
- Data Quality and Management
- Type
- article
- Field-Weighted Citation Impact
- 0.00