A refined methodological approach: Long-term stock market forecasting with XGBoost

Abstract One critical research gap that this study fills is artificial intelligence (AI) and machine learning applications that predict equity market index total returns using long-term prediction horizons, and by experimenting with Extreme Gradient Boosting (XGBoost). The presented models achieved significantly higher accuracy rates than the majority class rate, and they obtained better predictive scores in all metrics than the logistic regression model. The best-performing model had a 100% accuracy rate when negative returns were predicted with a p -value of 0.05121. The evidence from this study suggests that XGBoost, a neglected algorithm in the literature, can enable more empirically informed long-term portfolio management decisions regarding overall equity exposure. Moreover, a literature contribution of this study is a refined methodological approach for prospective studies when implementing binary classifiers of prospective stock market returns for enhanced real-life economic utility for investors. The constructed models generated probabilities of whether the S&P 500 will have positive or negative total returns, including dividend payouts, in the subsequent 12 months. The predictive metrics of these models were evaluated against traditional logit models and whether the accuracy rates statistically significantly exceeded the majority class rate.