Finite-sample Testing of Average Out-of-specification-rate Claims: A Learn-then-test Framework for Design-space Candidates

Abstract Purpose Predictive performance can identify a design-space candidate but cannot establish its quality claim. We developed a finite-sample procedure for a prespecified average out-of-specification (OOS) rate claim under a reference plan. Methods Construction data ranked operating conditions and defined a finite nested candidate family. Independent calibration data provided exact one-sided tests for the full candidates. Holm’s procedure controlled the family-wise error rate (FWER) at 0.05. The case study compared Parteck M100 and M200 on a finite grid of 315 measured setpoints per grade. Partial least squares (PLS) and random forest (RF) scores were evaluated on matched splits. Exact hypergeometric tests conditioned on the known OOS labels among construction setpoints. Results Among target candidate masses 0.10, 0.55, and 1.00 under the reference plan, the largest supported candidate contained 174 M100 setpoints and 32 M200 setpoints at a tolerated average OOS rate of 0.15. Matched outcomes showed a grade-associated difference before score modeling. PLS and RF preserved the grade ordering of supported mass, although membership differed, especially for M200. In simulation, empirical FWER for the full-candidate Holm procedure remained below 0.05, whereas the unadjusted comparator reached 0.105. The fixed M200 candidate was retained in M100; the fixed M100 candidate was not retained in M200. Conclusion The procedure supports an average OOS-rate claim for a candidate selected from a prespecified family under a stated reference plan. The procedure does not provide pointwise assurance. Candidate-family design, score-model choice, retention, and recertification must be interpreted separately.