Fast Sparse Learning from Streaming Data with LASSO

In this paper, we propose Online LASSO - a version of LASSO that is configured for streaming data. In standard LASSO, the penalty parameter is typically chosen by cross-validation, a procedure which requires the entire dataset upfront and repeated fitting. The main contribution of this work is in finding an easy and principled choice for the penalty parameter for every incoming data point, in cases where the input features are uncorrelated. The proposed Online LASSO has several benefits: i) it is memory and time efficient ii) it is easy to implement, iii) it does not require an initial batch of data to start, iv) it does not require any tuning (e.g., step size or tolerance), and finally v) it converges to the performance of the optimal predictor and correct selection of features. We demonstrate these capabilities and compare Online LASSO with standard LASSO as well as other adaptive LASSO variations and provide a discussion on their performances.