Scaling Laws in the Tiny Regime: How Small Models Change Their Mistakes

Neural scaling laws describe how model performance improves as a power law with size, but existing work has focused almost entirely on models above 100 M parameters. The regime below 20 million parameters, where TinyML and edge AI systems operate, remains largely unexamined. We train 90 models spanning 22 K to 19.8 M parameters across two architecture families (a plain ConvNet and MobileNetV2) on CIFAR-100, varying width while holding depth and training protocol fixed. Both architectures follow approximate power laws, with exponents of α=0.156 (ScaleCNN) and α=0.106 (MobileNetV2). However, the power law does not hold uniformly: local exponents decay with scale, and MobileNetV2 saturates at 19.8 M parameters (αlocal=0.006), hitting a data wall. The structure of errors also changes with scale. The Jaccard overlap between error sets of the smallest and largest ScaleCNN models is only 0.35; compression changes which inputs are misclassified, not merely how many. Small models develop a triage strategy, concentrating capacity on easy classes (Gini of per-class accuracy: 0.26 at 22 K params vs. 0.09 at 4.7 M) while effectively abandoning the hardest ones (bottom-5 class accuracy: 10% vs. 53%). The smallest models achieve the lowest ECE values (0.013 vs. peak 0.110 at mid-size), reversing the typical overconfidence–capacity relationship, though this partly reflects a global-mean matching artifact rather than well-calibrated per-bin confidence. On CIFAR-100, aggregate accuracy alone is therefore a misleading basis for edge deployment decisions; validation must happen at the target model size. All findings in this study are based on CIFAR-100 (32 × 32, 100 classes); their generalizability to other datasets, resolutions, and architectures remains to be verified.