Small Effects, Big Noise: A Methodological Audit of Module Evaluation in Small-Scale Crack Instance Segmentation

Abstract Crack instance segmentation papers built on YOLO-seg routinely claim gains by contrasting a baseline against the same model plus a proposed module, yet their statistical reliability is rarely examined. We audit a representative module, a skeleton-aware label assigner (STAL), across three epoch budgets, three seeds, three calibers, and eight metrics, and replicate on a second dataset and a second, structurally different module. Sweeping its tolerance proves the module is not inert; yet no gain is usable. The paired effect stays below 0.9 pt and flips sign with budget and caliber; the cross-seed correlation flips sign on identical runs; and the minimum detectable effect exceeds the observed one by 2.7–13.3×. All seven domain metrics are undecidable, one quantised to 11.1 pt by its 9 crossing pairs, and with three seeds exact Wilcoxon cannot return p < 0.25 at any effect size. Both replications reproduce the resolution ceiling, not the effect. From these results we derive an eight-rule minimum discernible protocol: effect-size floor, fixed caliber, verified pairing, MDE beside every difference, per-seed dispersion, fixed budgets, computed seed budget; every rule checkable from a single per-seed table.

Authors

Institutions

Publication Details

Journal
Research Square
Published
2026-11-16
DOI
https://doi.org/10.21203/rs.3.rs-11379838/v1
Primary Topic
Infrastructure Maintenance and Monitoring
Type
preprint

Funders

Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Small Effects, Big Noise: A Methodological Audit of Module Evaluation in Small-Scale Crack Instance Segmentation

Xin Cao
4 links
Research Square
Infrastructure Maintenance and Monitoring
preprint

Small Effects, Big Noise: A Methodological Audit of Module Evaluation in Small-Scale Crack Instance Segmentation

Xin Cao
preprint en

Abstract

Abstract Crack instance segmentation papers built on YOLO-seg routinely claim gains by contrasting a baseline against the same model plus a proposed module, yet their statistical reliability is rarely examined. We audit a representative module, a skeleton-aware label assigner (STAL), across three epoch budgets, three seeds, three calibers, and eight metrics, and replicate on a second dataset and a second, structurally different module. Sweeping its tolerance proves the module is not inert; yet no gain is usable. The paired effect stays below 0.9 pt and flips sign with budget and caliber; the cross-seed correlation flips sign on identical runs; and the minimum detectable effect exceeds the observed one by 2.7–13.3×. All seven domain metrics are undecidable, one quantised to 11.1 pt by its 9 crossing pairs, and with three seeds exact Wilcoxon cannot return p < 0.25 at any effect size. Both replications reproduce the resolution ceiling, not the effect. From these results we derive an eight-rule minimum discernible protocol: effect-size floor, fixed caliber, verified pairing, MDE beside every difference, per-seed dispersion, fixed budgets, computed seed budget; every rule checkable from a single per-seed table.

Research Square
Chongqing Jiaotong University (CN)
Chongqing Jiaotong University
Infrastructure Maintenance and Monitoring
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon