Bayesian-inspired verification of data quality rule candidates

Purpose Organisations rely on cataloged data quality rules for governance, yet profiler outputs still need screening before promotion to production monitors. This paper studies how semantic plausibility and SQL violation evidence can form an auditable verification score for candidate rules. Design/methodology/approach We present Route B, a Bayesian-inspired, prior-adjusted verification score fusing Laplace-smoothed SQL violation rates with hierarchical semantic prior templates. Evaluation uses TPC-H, Financial, and a desensitised Shenzhen open-data catalog (56 candidates). Shared candidate sets are compared with prior-only, SQL-only, confidence-interval, and beta-binomial baselines alongside Route B and optional closed-loop prior correction. Findings On the Catalog benchmark, prior-only yields F1 0.33 (36 false promotions) versus Route B F1 0.95 (2 false promotions), cutting proxy review time by about 510 min. Under ambiguous violation rates, hierarchical priors raise F1 from 0.539 to 0.800 versus likelihood-only filtering. On clean warehouses, SQL validation already separates most candidates, and Route B matches SQL-only at precision 1.000 on accepted rules. Research limitations/implications Evaluation uses synthetic warehouses and a desensitised catalog; candidates are scored independently; catalog labels follow a written steward protocol; mixed row noise up to 20% does not change warehouse F1. Practical implications Route B sits between rule discovery and monitors such as Great Expectations or Deequ as an audit-oriented acceptance layer. A reproduction package regenerates the reported tables without a live LLM endpoint. Originality/value An auditable prior-adjusted verification framework with closed-form accept boundaries; a warehouse-plus-catalog verification benchmark from SQL-profiler proposals; and evidence on when semantic–SQL fusion changes decisions relative to SQL-only or prior-only filtering.

Authors

Publication Details

Journal
Data Technologies and Applications
Published
2026-10-19
DOI
https://doi.org/10.1108/dta-07-2026-0785
Primary Topic
Data Quality and Management
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
article

Bayesian-inspired verification of data quality rule candidates

Lei Qiongyu, Hui Liu, Rui Feng
Data Technologies and Applications
Data Quality and Management
article

Bayesian-inspired verification of data quality rule candidates

Lei Qiongyu, Hui Liu, Rui Feng
article en

Abstract

Purpose Organisations rely on cataloged data quality rules for governance, yet profiler outputs still need screening before promotion to production monitors. This paper studies how semantic plausibility and SQL violation evidence can form an auditable verification score for candidate rules. Design/methodology/approach We present Route B, a Bayesian-inspired, prior-adjusted verification score fusing Laplace-smoothed SQL violation rates with hierarchical semantic prior templates. Evaluation uses TPC-H, Financial, and a desensitised Shenzhen open-data catalog (56 candidates). Shared candidate sets are compared with prior-only, SQL-only, confidence-interval, and beta-binomial baselines alongside Route B and optional closed-loop prior correction. Findings On the Catalog benchmark, prior-only yields F1 0.33 (36 false promotions) versus Route B F1 0.95 (2 false promotions), cutting proxy review time by about 510 min. Under ambiguous violation rates, hierarchical priors raise F1 from 0.539 to 0.800 versus likelihood-only filtering. On clean warehouses, SQL validation already separates most candidates, and Route B matches SQL-only at precision 1.000 on accepted rules. Research limitations/implications Evaluation uses synthetic warehouses and a desensitised catalog; candidates are scored independently; catalog labels follow a written steward protocol; mixed row noise up to 20% does not change warehouse F1. Practical implications Route B sits between rule discovery and monitors such as Great Expectations or Deequ as an audit-oriented acceptance layer. A reproduction package regenerates the reported tables without a live LLM endpoint. Originality/value An auditable prior-adjusted verification framework with closed-form accept boundaries; a warehouse-plus-catalog verification benchmark from SQL-profiler proposals; and evidence on when semantic–SQL fusion changes decisions relative to SQL-only or prior-only filtering.

Data Technologies and Applications
Openalex Percentile: Top 9%
Data Quality and Management
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.