Eligible Is Not Preferred: A Preregistered Recruiter-Panel Audit of an AI Hiring Ranker

Whether an AI hiring ranker agrees with recruiters depends on which recruiter judgment it is asked to recover. We report a preregistered audit of one frozen production LLM ranker on ten live requisitions. Four blinded professional recruiters assessed 150 candidate–requisition pairs (148 people), first for eligibility and then for suitability and advancement. The same judges gave strong or exceptional grades to 62 of 218 assessments they had marked ineligible, and advanced 20 of those candidates (20 of 171 selections). Under the study rubrics, the system's first pick was eligible in 7 of 9 requisitions with a resolved first-pick label and sat in the eligibility-constrained panel top set in 5 of 9 requisitions with a nonempty reference; a post-hoc reference built from majority advancement votes contained it in 3 of 9. These rates use different requisition sets. Raw eligibility agreement was low (α = 0.30). In held-out-judge comparisons that match prediction sizes within the judged pools, recruiter and system precision averaged 0.41 each against eligibility-constrained tiers but 0.41 and 0.52 against advancement votes, so the apparent parity depended on the reference. Twelve of the 33 eligibility-constrained top-set members came from outside the system's top five, and one was never scored by the system. The audit shows how the human reference, the candidate pool, and the prediction size shape measured agreement, and argues for interfaces that surface requirement evidence and record exceptions.