Citation traceability and inter-coder agreement in one scoping review: an exploratory audit

Abstract Background Agreement between extractors does not show whether their citations lead back to the supporting text, nor whether screening eligibility rules reach the fields later extracted. Methods We audited a scoping review by our team: 277 included reports (coded corpus 390) and 29 study-level fields. Co-primary measures were field-level proportion of agreement (Po) across the 29 fields and the whole-passage verbatim-anchoring rate of 9,415 citation judgments (coder A, pre-adjudication), with Krippendorff’s alpha (α) secondary. Citation checks establish relocation, not support. The registered screening protocol required the visual channel to be used in the pipeline of the evaluated model; that rule had not reached the extracted image-channel field, so we re-applied it. Two reviewers independently re-assessed 115 records (Cohen’s kappa = 0.83) with third-party arbitration; 275 coded records were not re-verified. Results Re-applying that pre-specified rule excluded 107 records, confirmed 2, and left 6 unresolved (excluded from synthesis; sensitivity 283), reducing the 390-record coded corpus to 277 included reports. Reliability and anchoring denominators remain the 390-record coded corpus, not 277 included reports. Eight of 29 fields fell below Po 0.80; of 26 fields with estimable α, 11 fell below 0.67, seven of those with Po ≥ 0.80. Three free-text fields (exact-match only) ranged from 0.04 to 1.00. Of 9,415 citation judgments, 6,215 (66.0%) were whole-passage verbatim, 2,229 (23.7%) first-25-character only, and 971 (10.3%) unmatched; a tolerant search relocated 946 without verifying support. Gate 1 returned 15/23 (65.2%) verbatim re-hits, below the ≥ 90% threshold, and stopped Gate 2; Gate 3 returned Po = 0.6435 across the 115-row attribution ledger (115 attribution-judgment cells; 74/115). Conclusions This audit shows the instrument built to classify extraction failures failed its own checks in the way it was designed to detect; agreement and citation anchoring need separate checks. Five operationalisations (field-type classification, free-text exclusion, floors, 26-field grouping, prevalence-adjusted analyses) and the co-primary measures were set after the data were visible. These exploratory results do not establish the correctness of extracted values or support a clinical conclusion.