AI assistant Skills for Self-Auditing: A Two-Layer Verification System for Data and Semantic Claims, Demonstrated on Twenty-One Academic Papers

A research paper written with the assistance of an AI agent can be wrong in two ways. A printed number may fail to match the data it came from, and the claim about the data may drift even when every number is right. We present a two-layer claim-verification system, delivered as reusable, open-source AI skill files that any researcher can run on their own manuscript with the AI assistants they already use. The data layer ties every printed number to a frozen copy of its source file and re-checks the tie in both directions by script; the semantic layer checks the sentences themselves — comparisons, causal claims, stated scope — against the same data sources. We applied it first to our own six papers, where it registered roughly sixty-four defects, at least twenty-four of which were visible only to the semantic layer. We then ran it on fifteen published academic papers with open data. A blind audit of a paper already corrected by its journal caught that correction and one the journal had missed. Eleven of the fifteen papers carried substantive defects, whether or not their authors disclosed AI assistance in writing. And for many papers, most of what is claimed cannot be checked at all from public materials — a problem this system makes measurable. We report the defect types the system finds, the limits of what it can show, and how to run it.