The Impact of Data Distribution and Data Noise on the Fitting and Prediction Performance of Regression Models

Regression analysis is an important technique in data analysis field. Nowadays, diverse of regression models have been proposed. However, when dealing with different data sets, the selection of regression models has no gold standard, which is often determined by the experience of the modeler and lacks universality. This study aims to fill the gap of regression methods selection based on biased and noisy data. To better applying data sets with different levels of skewness and noise, simulated data was generated. This paper tested the effects of models on three typical regression models: linear regression, ridge regression and lasso regression. Specifically, this paper firstly explored the effects of data distribution on these models. By simulating a series of normal distributed data sets with skewness parameter from -20 to 20, which has found that when the data was biased, the linear regression had the best performance, followed by lasso regression and ridge regression. Next, this paper generated different levels of noisy data sets and tested their fitting and prediction performance on regression models. The results show that both linear regression and lasso regression performance well, while linear regression has a little better performance with slight noisy data. In contrast, the robustness of ridge regression is not as good as two other models. Overall, this paper studied the impacts of different types of data on regression models, and provided new insights for model selection based on different types of datasets.