Bayesian-inspired verification of data quality rule candidates
Purpose Organisations rely on cataloged data quality rules for governance, yet profiler outputs still need screening before promotion to production monitors. This paper studies how semantic plausibility and SQL violation evidence can form an auditable verification score for candidate rules. Design/methodology/approach We present Route B, a Bayesian-inspired, prior-adjusted verification score fusing Laplace-smoothed SQL violation rates with hierarchical semantic prior templates. Evaluation uses TPC-H, Financial, and a desensitised Shenzhen open-data catalog (56 candidates). Shared candidate sets are compared with prior-only, SQL-only, confidence-interval, and beta-binomial baselines alongside Route B and optional closed-loop prior correction. Findings On the Catalog benchmark, prior-only yields F1 0.33 (36 false promotions) versus Route B F1 0.95 (2 false promotions), cutting proxy review time by about 510 min. Under ambiguous violation rates, hierarchical priors raise F1 from 0.539 to 0.800 versus likelihood-only filtering. On clean warehouses, SQL validation already separates most candidates, and Route B matches SQL-only at precision 1.000 on accepted rules. Research limitations/implications Evaluation uses synthetic warehouses and a desensitised catalog; candidates are scored independently; catalog labels follow a written steward protocol; mixed row noise up to 20% does not change warehouse F1. Practical implications Route B sits between rule discovery and monitors such as Great Expectations or Deequ as an audit-oriented acceptance layer. A reproduction package regenerates the reported tables without a live LLM endpoint. Originality/value An auditable prior-adjusted verification framework with closed-form accept boundaries; a warehouse-plus-catalog verification benchmark from SQL-profiler proposals; and evidence on when semantic–SQL fusion changes decisions relative to SQL-only or prior-only filtering.
Authors
- Lei Qiongyu
- Hui Liu (ORCID: https://orcid.org/0009-0005-5987-1040)
- Rui Feng
Publication Details
- Journal
- Data Technologies and Applications
- Published
- 2026-10-19
- DOI
- https://doi.org/10.1108/dta-07-2026-0785
- Primary Topic
- Data Quality and Management
- Type
- article
- Field-Weighted Citation Impact
- 0.00