Admissibility of Gold-Free Evaluation Signals for Clinical Information Extraction: A Sealed Benchmark Study

Objective. In most clinical information-extraction settings there is nolabeled ground truth, so systems are developed and trusted on label-freesignals such as self-consistency, one model judging another, and multi-modelconsensus. We test whether such a signal actually predicts true accuracy, aproperty we call admissibility, on a benchmark that has gold labels butwithholds them until the end. Materials and Methods. Using the public GENIA biomolecular-event extractiontask, we developed a large language model extraction pipeline under a strictseal: the developer saw only the task definition and unlabeled text, with nogold, no prior results, and no outside literature. Every keep-or-discarddecision was driven by a decomposed, multi-model evaluation signal that wefirst calibrated without gold by mutation testing, planting known errors andconfirming the signal detected them on the correct axis. We froze 16 pipelinevariants with the label-free verdict each received, then revealed gold onceand scored every variant with the official metric, pre-registering a 0.70rank correlation as the bar for admissibility. Results. The label-free pipeline reached F1 31 against 17 for a same-budgetsingle-prompt baseline, a gain of 14 points, so competent extraction withoutlabels is achievable. However, the evaluation signal was not admissible. Rankcorrelation between the label-free proxy and gold across the 16 variants was0.54, below the pre-registered 0.70 bar. The signal was precision-biased: itunder-counted missed events and rewarded producing fewer items. A sequence ofrefinements that the signal scored as steady improvements, with the proxyrising from 0.70 to 0.80, in fact reduced true F1 from 31 to 28. Restoring aset of events the signal had rejected raised true accuracy, confirming thebias. Discussion. A label-free signal can be calibrated, discriminative, and stillfail to rank systems the way ground truth does. The failure mode is specificand detectable: cross-model adjudication of omissions, mutation testing of theevaluation metric itself, and a one-time reveal where gold exists arepractical safeguards. Conclusion. Admissibility should be tested rather than assumed before alabel-free signal is trusted to govern a clinical extraction pipeline.