When the Semantic Layer Is Wrong: Do LLM Data Agents Follow or Flag Faulty Business Definitions?

Semantic layers improve text-to-SQL accuracy when their business definitions are correct. We study 24 controlled faults in an Apache Ossie model over TPC-DS, covering formulas, filters, synonyms, fiscal calendars, join keys and value mappings. Each fault has a coherent form, whose documentation agrees with it, and a contradictory form, whose documentation retains the reference definition. In 1,512 single-turn, tool-free trials with three models, a correct layer raised accuracy from 60–71% to 100% on this benchmark. Coherent faults were followed in 46–73% of trials and reduced pooled accuracy relative to no layer by 26 percentage points (95% CI, 9–43). Contradictory faults were located in 92–100% of trials but followed in 4–33%. Some responses raised a concern about the faulted definition and still used it. A generic audit instruction did not reliably reduce following and raised concerns on the correct layer by 17 percentage points. Plausible business conventions may require an independent policy reference to verify. Author annotations of 417 sampled concerns gave agreement with the detection rule of Cohen’s κ = 0.89 for layer concerns and 0.90 for located faults.