The Protocol is the Result: How Evaluation Choices Flip the Verdict on Lightweight Image Restoration

Lightweight image restoration is infrastructure: student-scale super-resolution and low-light enhance-ment models are re-implemented, compared and reported on constantly. Their reported numbers are, however, not directly comparable, because the evaluation protocol is rarely stated in full. We decompose the protocol into five operational dimensions and measure each of them on a matrix of 21 protocol cells across three tasks and five random seeds (105 evaluations in total). On the ×4 super-resolution task the choice of protocol moves the reported metric by 3.889 dB—and by 19.794 dB once best-of-60 cropping is included—which is 4.98 times the 0.782 dB gain the trained model achieves over its bicubic reference. The effect is not statistical noise: seed-to-seed variation accounts for 0.49–0.93%of the protocol span on that task. Switching off the antialias flag of the downsampling operator—a single boolean, with the same weights, the same images and the same code path—moves the model’s gain over bicubic from +0.818 dB to −1.753 dB, reversing the verdict. We also find that the claim is conditional rather than universal: on low-light enhancement the protocol span is only 0.21 times the training gain, because that task has far more headroom. Finally, a coding of 63 recent papers shows that only one states its aggregation method, none states its validation batch size, and of the 15 papers whose validation and test sets can be judged from the text, none states whether those sets coincide (the other 48 were not coded, since deciding it needs the code or the appendix). We close with six reporting requirements that would remove the ambiguity.