Policy-As-Code Governance For Bronze-Layer Ingestion In Regulated Databricks Medallion Lakehouse Architectures

Modern data platforms widely use the medallion architecture to organize pipelines on lakehouse systems such as Databricks. Governance controls, including schema validation, sensitive-data handling, retention, lineage tracking, quality checks, and access management, are typically applied in downstream layers or through manual processes. This leaves the bronze layer, where raw data first enters the platform, inconsistently governed. In regulated sectors, a lapse at this layer, such as an unhandled schema change or unmasked sensitive field, can spread across the pipeline and create compliance, financial, and operational risk, because downstream analytics and machine learning depend on the integrity of this raw foundation. This paper reviews and synthesizes existing governance, data quality, and policy-as-code approaches, then proposes a unifying framework that expresses governance requirements as declarative, version-controlled policies enforced automatically at bronze ingestion. The framework combines a policy repository, an enforcement engine with allow, warn, quarantine, and reject outcomes, and an append-only audit store, using Databricks-native tools and general policy engines. The synthesis shows that six governance concerns, namely schema, sensitive data, retention, lineage, quality, and access, can be coordinated by a single policy layer in which each concern is handled by the mechanism suited to it: an ingestion-time gate for schema, sensitive data, and quality; a lifecycle policy for retention; metadata capture for lineage; and access control applied at query and access time. The contribution is a unified policy layer that coordinates these six concerns with continuous auditing, rather than forcing all of them through one enforcement point. Novelty is claimed relative to the approaches examined in this review.

Authors

Publication Details

Journal
International Journal of Advanced Artificial Intelligence Research
Published
2026-10-27
DOI
https://doi.org/10.55640/ijaair-v03i10-03
Primary Topic
Data Quality and Management
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
article

Policy-As-Code Governance For Bronze-Layer Ingestion In Regulated Databricks Medallion Lakehouse Architectures

Shikha Sharma
International Journal of Advanced Artificial Intelligence Research
Data Quality and Management
article

Policy-As-Code Governance For Bronze-Layer Ingestion In Regulated Databricks Medallion Lakehouse Architectures

Shikha Sharma
article en

Abstract

Modern data platforms widely use the medallion architecture to organize pipelines on lakehouse systems such as Databricks. Governance controls, including schema validation, sensitive-data handling, retention, lineage tracking, quality checks, and access management, are typically applied in downstream layers or through manual processes. This leaves the bronze layer, where raw data first enters the platform, inconsistently governed. In regulated sectors, a lapse at this layer, such as an unhandled schema change or unmasked sensitive field, can spread across the pipeline and create compliance, financial, and operational risk, because downstream analytics and machine learning depend on the integrity of this raw foundation. This paper reviews and synthesizes existing governance, data quality, and policy-as-code approaches, then proposes a unifying framework that expresses governance requirements as declarative, version-controlled policies enforced automatically at bronze ingestion. The framework combines a policy repository, an enforcement engine with allow, warn, quarantine, and reject outcomes, and an append-only audit store, using Databricks-native tools and general policy engines. The synthesis shows that six governance concerns, namely schema, sensitive data, retention, lineage, quality, and access, can be coordinated by a single policy layer in which each concern is handled by the mechanism suited to it: an ingestion-time gate for schema, sensitive data, and quality; a lifecycle policy for retention; metadata capture for lineage; and access control applied at query and access time. The contribution is a unified policy layer that coordinates these six concerns with continuous auditing, rather than forcing all of them through one enforcement point. Novelty is claimed relative to the approaches examined in this review.

International Journal of Advanced Artificial Intelligence Research
Openalex Percentile: Top 9%
Data Quality and Management
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon