Machine Learning Methodologies for the Systematic Prediction of S&P 500 Index Deletions

This article aims to develop a methodology to systematically predict discretionary deletions from the S&P 500 Index before a rebalance is announced. Using the methodology from the S&P 500 along with past predictive studies, the author creates 93 unique features across market capitalization, trading, idiosyncratic volatility, and financial statement segments. He uses these features with logistic regressions, random forests, and XGBoosted random forest models to predict deletions from the index. Features are selected using two different methodologies, one using principal component analysis and a random forest classifier (RFC) and the other with solely an RFC. Event studies show statistically significant returns at 95% confidence with an average cumulative abnormal return of −6.16% to the announcement date for deleted securities starting four days before announcement. A significant improvement in predictive power and predictive metrics for both random forest and XGBoosted random forest methodologies is observed compared to that of the logistic regression. Average true positive predictions across models increased from 28 of 37 with logistic regression, to 31 of 37 with random forests, and up to 34 of 37 with XGBoosted random forests. The best model, using a random forest model with the feature set “pca_50,” accurately predicted 32 of 37 discretionary deletions, exhibiting the highest precision and F1 scores in comparison to all alternative models under consideration. All final models were statistically significant at 99% when evaluating their performance and predictive metrics in a permutation test, thus challenging the strong-form efficient market hypothesis and indicating validity in predicting deletions from the S&P 500 before the reconstitution is announced.