Transdimensional Model Learning With Online Feature Selection Based on Predictive Least Squares

Our interest in obtaining predictive models from vast volumes of data has never been greater. Often, we want these models to be as simple as possible but not simpler. Reducing the complexity or size of a model is accomplished by enforcing sparsity in the model’s parameters or structure. In the context of sparse regression, LASSO and its variants have become standard practice, but they rely on the choice of the penalty parameter for desired degrees of sparsity or predictability. In this work, we develop a novel online inference approach for transdimensional model learning with online feature selection, built on the fundamental principles of predictive least squares. The solution path follows the predictive error at each step, prioritizing more predictive features in the model. We provide a variety of examples to analyze the capabilities of the proposed method and evaluate its performance against both standard and recently proposed feature selection methods.