How Much Data Is Enough? Conformal Sample-Size Planning from Pilot Learning Curves, with a Re-Analysis of the Learning Curve Database
Deciding how many labelled examples to collect is one of the most common questions in applied machine learning, yet it is usually answered with rules of thumb or by extrapolating a fitted learning curve without any statement of uncertainty. We revisit the Learning Curve Database (LCDB 1.0) of Mohr et al. and its public replication package. We first reproduce its curve-fitting benchmark and confirm that saturating four-parameter models, led by MMF4, extrapolate accuracy best. We then pose the inverse question a practitioner actually asks: given a pilot learning curve up to 128, 256 or 512 samples and a target accuracy, how many training samples are needed? On 9,445 planning problems from 1,414 curves, 201 datasets and 10 learners, with ground truth taken from independent test splits, the best extrapolators are the worst planners: MMF4 and WBL4 declare a reachable target unreachable in 50.2% and 65.1% of cases. Popular rules of thumb such as 10 samples per feature are essentially uncorrelated with the data actually required (Spearman rho = 0.03). We propose LC-Plan, which meta-learns a mapping from pilot-only features to the required sample size across datasets and wraps it in conformalized quantile regression. LC-Plan reduces the median planning error to a factor of 1.50 (versus 2.12 for the best single curve model and 5.25 for the best rule of thumb), is within a factor of two of the truth in 66.7% of cases, and ranks first in a Friedman test over 177 datasets (p < 10^-10). Its nominal 90% intervals achieve 89.7% coverage, whereas bootstrap intervals around fitted curves cover only 47.0% to 53.5%. We release LC-Plan as an open-source Python package that returns a sample-size estimate and interval from a scikit-learn estimator and a pilot dataset in one call.
Authors
- Nezar Abdilah Prakasa
Publication Details
- Journal
- Zenodo (CERN European Organization for Nuclear Research)
- Published
- 2026-11-17
- DOI
- https://doi.org/10.5281/zenodo.24303799
- Primary Topic
- Machine Learning and Algorithms
- Type
- preprint