Auditing Qualification Validity Arguments under External-Transparency Constraints: A Replicable Methodological Framework

Abstract Evaluators can ask whether an assessment is valid. They have no systematic way to ask a prior question: do aqualification's own governing documents carry evidential support for each inference in its validity argument,before any assessment instrument is examined? This paper presents a replicable documentary procedure thatanswers it. Under Kane's argument-based framework, four sequential inferences, Scoring, Generalisation,Extrapolation, and Implication, each require independent support, and a gap in any one cannot be repaired byevidence bearing on another. The procedure codes governing documents as Explicit, Implicit, or Absent foreach inference. An inference can be recorded as carrying no evidence at all, rather than forced onto a scale ofstrength. An Absent rating is not a verdict on assessment quality; it is a transparency finding that locateswhere the public record does not publish the evidential basis the argument requires. Three documents fromthe New Zealand Diploma in Business (Level 5, Accounting strand, Ref. 2459) demonstrate the procedure atits hardest decision points: the Implicit/Absent boundary, cross-document pattern reading, and the authoritypresent, relevance-absent distinction. The low-resource procedure applies to any qualification governedthrough public documents, and further applications are needed to establish inter-rater reliability and crossqualification consistency. Keywords: validity argument; competency-based qualification; Kane's framework; documentary analysis;external-transparency constraints