KLSampler: A Kullback-Leibler Divergence Based Undersampling Technique for Machine Learning Applications

Class imbalance poses a significant challenge in binary classification tasks, often degrading model performance and limiting generalization, particularly in domains where minority classes are critical. Traditional undersampling techniques tend to remove majority class samples without considering their distributional characteristics, which may lead to information loss and skewed representations. In this study, we propose KLSampler, an undersampling algorithm based on Kullback–Leibler (KL) divergence, designed to address this limitation by preserving the distributional integrity of the majority class. The algorithm begins with cluster-aware proportional sampling and iteratively refines the selected subset by minimizing the KL divergence between the original and sampled majority class distributions, guided by kernel density estimation. Experimental evaluations on both real-world and synthetic datasets demonstrate that KLSampler achieves improved class balance while maintaining low KL-divergence and minimal covariate shift. These outcomes are validated using both divergence-based and domain classifier metrics. Overall, the results suggest that KLSampler offers a robust and distribution-aware alternative to conventional undersampling techniques, providing improved data quality and supporting more reliable model training in imbalanced learning scenarios.