When the Baseline Also Abstains: Pitfalls in Evaluating Abstention for Document Audit

Retrieval-augmented (RAG) auditors invent disagreements: asked for a value the document never gives, they answer anyway, producing false-positive divergences. We build an ontology-gated auditor that abstains on type/scope mismatch — and, more importantly, we find that measuring whether such a system abstains well is itself error-prone. Using a benchmark of Brazilian contracts and government tender notices, we surface three pitfalls, each illustrated by a concrete mistake we made and corrected: (i) a single-sample illusion, where a rare false-positive rate reads as "0%" until bootstrapped, and a self-consistency "free lunch" turns out to be sampling noise; (ii) that "hard" is defined by the model, not the taxonomy — a naive baseline ties an abstaining system not only on wholly-missing concepts but even on taxonomically-adjacent ones, so a gate helps only where the model actually conflates a concept pair; and (iii) a keyword-absence annotation trap, where labeling ABSTAIN from a missing keyword is invalid on real legal text that paraphrases concepts. We distill the lessons into a checklist for evaluating abstention in document audit, and release the system, benchmark, and evaluation harnesses.