Using Sequential Design of Experiments and Non‐Uniform Space‐Filling Designs for Improved Model Selection

ABSTRACT The discussion of model selection often starts with the data already collected and the analyst considering how to choose between the best of some already identified model choices. This paper discusses some opportunities for utilizing sequential experimentation to help discern between model choices by collecting additional data in regions of the input space where there are larger differences in the predicted responses between the models being compared. We show how augmenting an existing design with a non‐uniform space‐filling design, which has been previously used to target regions of the input space closer to the optimum or where prediction is most uncertain, can be adapted to improve understanding of differences in model performance as well as help to quantify these differences. The new method is illustrated with examples that compare nested regression models as well as unnested linear and non‐linear models. Some graphical and numerical summaries are used to discern the model performance with a new metric, self‐validated mean squared prediction error, proposed to evaluate the predictive ability of a fitted model.