Small Effects, Big Noise: A Methodological Audit of Module Evaluation in Small-Scale Crack Instance Segmentation
Abstract Crack instance segmentation papers built on YOLO-seg routinely claim gains by contrasting a baseline against the same model plus a proposed module, yet their statistical reliability is rarely examined. We audit a representative module, a skeleton-aware label assigner (STAL), across three epoch budgets, three seeds, three calibers, and eight metrics, and replicate on a second dataset and a second, structurally different module. Sweeping its tolerance proves the module is not inert; yet no gain is usable. The paired effect stays below 0.9 pt and flips sign with budget and caliber; the cross-seed correlation flips sign on identical runs; and the minimum detectable effect exceeds the observed one by 2.7–13.3×. All seven domain metrics are undecidable, one quantised to 11.1 pt by its 9 crossing pairs, and with three seeds exact Wilcoxon cannot return p < 0.25 at any effect size. Both replications reproduce the resolution ceiling, not the effect. From these results we derive an eight-rule minimum discernible protocol: effect-size floor, fixed caliber, verified pairing, MDE beside every difference, per-seed dispersion, fixed budgets, computed seed budget; every rule checkable from a single per-seed table.
Authors
- Xin Cao (ORCID: https://orcid.org/0009-0006-4804-8742)
Institutions
- Chongqing Jiaotong University (CN)
Publication Details
- Journal
- Research Square
- Published
- 2026-11-16
- DOI
- https://doi.org/10.21203/rs.3.rs-11379838/v1
- Primary Topic
- Infrastructure Maintenance and Monitoring
- Type
- preprint
Funders
- Chongqing Jiaotong University