Balancing Accuracy and Reliability: Addressing Class Imbalance, Outlier Handling, and Data Leakage in Water Potability Prediction

Machine learning (ML) offers a fast alternative to traditional water testing, enabling efficient predictions of water potability. This study examined the effects of outlier handling and data leakage on model performance using a highly imbalanced potable water dataset balanced with SMOTETomek. Four pre-processing approaches were tested: dropping both missing values and outliers, retaining outliers but removing missing values, training on a cleaned training set with missing values dropped and using median imputation for the test set, and replacing missing values in the training set using MICE while retaining outliers. Results showed that the first two approaches, which preprocessed training and test sets, led to data leakage and inflated accuracy. Retaining outliers in the second approach increased accuracy, as outliers captured critical water quality variance. The third approach, avoiding leakage by training only on complete data and using median imputation in the test set, achieved higher and most reliable accuracy (68.10%), outperforming the fourth, where MICE imputation in training reduced test accuracy. Among trained models, including Random Forest (RF), AdaBoost, Gradient Boosting (GB), XGBoost (XGB), CatBoost (CB), and LightGBM (LGBM). CatBoost achieved the highest accuracy. These findings underscore the importance of balanced data and robust preprocessing in developing ML models for water quality assessment.