Weighted Gradient-Forest Imputer: Explainable and Scalable Missing Value Imputation for Workforce Analytics

Missing data is a critical issue in data analysis, undermining insights and predictions by distorting patterns and introducing biases. In employee datasets, missing entries complicate productivity assessments, pay equity analyses, and operational efficiency. Traditional imputation techniques often fail to capture complex data relationships, leading to biased results, while more advanced techniques, though accurate, tend to be computationally intensive and lack transparency. To address these challenges, a novel weighted Ensemble imputation technique integrating LightGBM and MiceForest named Weighted Gradient-Forest Imputer (WGFI) has been proposed. Six other existing imputation algorithms have also been implemented for comparison. Classification performance was evaluated with RFC, SVC, and NBC after imputation. Evaluations on datasets of 1,197 and 10,000 records show that the proposed method outperforms others, achieving an MAE of 4.17 and RMSE of 47.31 on the larger dataset, compared to MAE of 15.61 and RMSE of 143.45 on the smaller one. On the larger dataset, RFC paired with WGFI achieved 99.57% accuracy and an F1 score of 99.57%, compared to 96.76% accuracy on the smaller dataset. LIME has been used to analyze WGFI, providing transparent insights into feature contributions. These findings highlight the scalability, robustness, and interpretability of the proposed method, establishing it as a valuable tool for workforce analytics and data-driven decision-making.