Modified tree-based selection in hierarchical mixed-effect models with trees: A simulation study and real-data application

Hierarchical mixed-effects models with three trees-3Trees models-are a new advanced statistical learning approach in mixed-effect modeling. These methods utilize the classification and regression trees (CART) algorithm to select the best tree through a backfitting algorithm. However, this algorithm relies on a greedy approach, making the trees prone to overfitting, biased in split selection, and often far from the optimal solution, ultimately affecting model performance. Two novel methods are proposed-3Trees-EvTree and 3Trees-CTree-to address these limitations. The proposed methods are compared with the available methods through several simulation exercises in different settings and real datasets. The simulation study confirms that the 3Trees-EvTree method performs well compared to the previous method in terms of parameter estimation and prediction accuracy under clusMSE and clusPMSE. Meanwhile, the 3Trees-CTree model performs well in low-correlation scenarios and the semilinear function. In addition, the proposed methods also reveal that the results of actual application confirm their superiority over other competing methods. Some highlights of the proposed method are:•3Trees-EvTree and 3Trees-CTree model to improve prediction accuracy and to reduce bias of 3Trees model are presented•MSE, ClusMSE, PMSE, ClusPMSE, and bias criteria are used to evaluate model performance•Applied to estimate and predict household expenditure per capita dataset.