Linear Decision Trees: A Comparative Study with Insights on ReLU Neural Networks

Traditional decision trees also known as vanilla decision trees, use axis-aligned splits and are interpretable. Alternate implementations known as linear decision trees use oblique splits to form shorter trees that can improve feature utilisation and performance. In this paper, we compare linear decision trees with vanilla decision trees, random forests, and ReLU neural networks. We theoretically demonstrate that any ReLU neural network can be represented as a linear decision tree in a binary classification setting. We evaluate the performance of the four model classes using both generated and real-world datasets and assess the impact of noise, ground truth complexity, and dataset size. Our findings indicate that vanilla decision trees and random forests are well-suited for real-world tabular data, while linear decision trees perform better on datasets with less noise and linear ground truths. In contrast, neural networks excel with both linear and non-linear ground truths in several scenarios. Additionally, we analyse the interpretability of linear decision trees in comparison with vanilla decision trees and also discuss their suitability for interpreting neural networks.