Empirical Evaluation of Learning Curve Cross-Validation for Efficient Model Selection

Model selection in machine learning is commonly performed using cross-validation, where candidate models are evaluated to estimate their generalization performance. Although reliable, this approach becomes computationally expensive when many models must be evaluated repeatedly on large datasets. Learning Curve Cross-Validation (LCCV) addresses this issue by evaluating models on progressively larger subsets of data and pruning weak candidates early. In this work, we implement the LCCV algorithm and evaluate it on several classification tasks. The implementation estimates performance at different training sizes using repeated cross-validation with confidence intervals. Models are pruned using optimistic learning curve extrapolation, while the Morgan–Mercer–Flodin (MMF) model is used to skip intermediate evaluation points. Experiments on real-world and synthetic datasets compare LCCV with traditional full cross-validation in terms of runtime, model selection agreement, and pruning behavior. Results show that LCCV can prune many candidate models and significantly reduce runtime on larger datasets, while introducing some overhead on smaller datasets.