Visualizing Multi Class Decision Boundaries of Ensemble Tree Models for Improved Interpretability

Accurate and interpretable multi-class classification remains a significant challenge in machine learning, particularly for datasets with overlapping feature distributions. Traditional ensemble methods, such as Random Forest and boosting algorithms, often face a trade-off between accuracy and interpretability in Random Forests provide stability but may retain bias, while boosting models achieve high accuracy at the expense of fragmented and less understandable decision boundaries. The Hybrid Boosted Forest (HBF) is a novel ensemble framework that integrates the diversity of Random Forests with the adaptive weighting mechanism of boosting. HBF incorporates dynamic tree depth selection based on feature heterogeneity, weighted aggregation of tree predictions, and a controlled boosting stage that emphasizes misclassified samples, resulting in robust performance and interpretable decision boundaries. Evaluation of HBF on the Iris dataset using multiple feature pairs demonstrates superior performance compared with six state-of-the-art models, including Decision Tree, Random Forest, Extra Trees, AdaBoost, Gradient Boosting, and XGBoost. HBF achieves an accuracy of 98.1%, surpassing the next best model (XGBoost at 97.2%), while maintaining high interpretability (7/10) and balanced computational efficiency. Decision boundary visualizations illustrate smooth, structured, and human-understandable class separations compared with baseline models. The results confirm that HBF offers a robust, explainable, and computationally practical solution for multi-class classification, providing a promising direction for ensemble learning research that demands both performance and interpretability.