Analyzing Inconsistencies Across Financial Services Machine Learning Algorithms and Implementations

Various machine learning algorithms are being used for financial valuation and classification tasks. However, incon-sistency, which refers to the difference in output behaviors, has yet to be investigated across algorithms used for a particular financial task and across different implementations. We analyze inconsistencies across Multivariate Linear Regression and Feed-forward Neural Networks for financial valuation tasks on 11 datasets, and inconsistencies across Gradient Boosted Decision Trees and Random Forests for financial classification tasks on 12 datasets. We also analyze inconsistencies across two toolkits used to implement each algorithm and find statistically significant evidence for inconsistency across the algorithms and their imple-mentations. Our findings suggest that inconsistencies can vary based on the training dataset. Overall, training datasets should be analyzed for inconsistency during the selection process, especially in situations where multiple algorithms and implementations may be used for a particular task and agreement in output data is desired.