What Breaks in Small Generative Process Verifiers: Verdict Aggregation, Calibration Limits, and Discriminative–Generative Cascades
We study a 1.5B generative process verifier (DeepSeek-R1-Distill-Qwen-1.5B with a fixed step-judgment format) on PRM800K and find that its failures lie mostly where accuracy numbers do not point. (1) Step accuracy is a base-rate mirage. A trivial all-"correct" judge scores 0.904 on the test steps; our SFT verifier scores 0.897 while flagging 15 of 1,857 steps, one correctly. Its verdicts mirror the majority class, and neither GRPO (every sampled group agrees, so advantages vanish) nor decode-time derivation repairs this. Training on multipath-consistent traces does (solution accuracy 0.417 → 0.583, McNemar p ≈ 10^-5). (2) The verifier knows more than it says. Verdict-token probabilities reach step AUROC 0.81, and noisy-OR aggregation over them gives 0.883 solution accuracy at ECE 0.092 with no fitting (SFT v2, 103 test traces), while the emitted confidence is two-valued by target design. (3) Step-label interference is real but small. Teacher-forcing an "incorrect" judgment at step k leaves later hard verdicts unchanged (CIs exclude drag increases above 0.4 points) but raises later steps' P(incorrect) by about one probability point (SFT v2: +0.9, 95% CI [+0.1, +1.9]). (4) Cascades trade cost for accuracy. Escalating only the traces a discriminative PRM is unsure about to the token-probability readout reaches 0.820 accuracy on 891 traces (the 103 test traces plus 788 assembled from their candidate pools) while calling the generative verifier on 20.5% of them. This beats the discriminative PRM alone (0.744; +7.7 points, problem-clustered 95% CI [+3.6, +12.3]) and matches running the readout on every trace within noise (0.773; +4.6 points, CI [-1.7, +11.4]) at a fifth of its generative cost.
Authors
- Qiwei Li
Publication Details
- Journal
- Zenodo (CERN European Organization for Nuclear Research)
- Published
- 2026-10-28
- DOI
- https://doi.org/10.5281/zenodo.23719740
- Primary Topic
- Adversarial Robustness in Machine Learning
- Type
- preprint