Evaluating the Effect of Tree Depth on the Performance of Decision Tree Classifiers

The maximum depth of a decision tree is one of the most influential hyperparameters governing the trade-off between model bias and variance. A tree that is too shallow may fail to capture the underlying structure of the data (underfitting), whereas a tree that is too deep tends to memorize training examples, harming its ability to generalize (overfitting). This paper presents a systematic empirical evaluation of the effect of maximum tree depth on classification performance using the CART-based DecisionTreeClassifier implementation from scikit-learn. Experiments are conducted on two widely used benchmark datasets — the Breast Cancer Wisconsin (Diagnostic) dataset and the Wine dataset — with depth values ranging from 1 to 20. For each depth, we report training accuracy, held-out test accuracy, and 5-fold stratified cross-validation accuracy, alongside structural complexity metrics such as the number of leaves and total nodes. Results show that test accuracy rises sharply for shallow trees, peaks at a moderate depth, and then plateaus or slightly declines as depth increases further, while training accuracy converges to a perfect fit. On the Breast Cancer dataset, cross-validation accuracy peaked at a depth of five, while training accuracy reached 100% by depth six, clearly illustrating the onset of overfitting. These findings reinforce the practical guidance that tree depth should be tuned via cross-validation rather than fixed a priori, and that pruning or depth-limiting strategies are essential for building decision trees that generalize well.