Flagging stable classifier errors with an inventory-mediated LLM judgement: an exploratory study and a preregistered replication

Errors that a classifier repeats under paraphrase and reordering of its input ("stable errors") escape detectors that look for inconsistency among the model's own outputs. We asked whether disagreement between a classifier and an LLM judgement routed through an explicit external structure (a WordNet sense inventory for word-in-context judgements; atomic facts and label definitions for NLI) flags such errors without labels, and how much of any gain needs the structure rather than a second LLM judgement. Study 1 (WiC test, ChaosNLI) became exploratory when a preregistered paraphrase check failed; Study 2, a preregistered replication on MCL-WiC, stopped under its own rule (91 stable errors < 100). Disagreement with the inventory-mediated judgement flagged stable errors more often than the classifier's confidence in all seven word-in-context configurations (DeBERTa × Qwen: 0.206 [0.142, 0.268] in Study 1, 0.268 [0.161, 0.374] in Study 2), but not consistently on all errors; how much of that advantage needed the inventory could not be pinned down, because a working unstructured second judgement reached between 25% and 78% of it (post hoc). Part of the structured judge's edge corrected the unstructured judge's bias towards "different"; against a label-free debiased judge the edge remained on WiC test, whose reference derives from the same inventory, and on MCL-WiC it depended on the correction (post hoc). Two independent raters confirmed the reference on only 100 of Study 1's 208 stable errors. On NLI the structured judgement was worse than the unstructured one. We report both preregistrations, deviations and the stop decision. Preprint, under review. The preregistrations (OSF) are under embargo and will be made public on publication; their ids, times and hashes are listed in Supplementary S1. Code, configuration, tables and aggregate outputs: doi:10.5281/zenodo.24132227.