Improving Online Payment Fraud Detection with Feature Engineering and Cost-Sensitive Machine Learning

Financial fraud detection is a critical and challenging task, especially in the context of large-scale transaction datasets that exhibit high class imbalance and complex behavioral patterns. In this study, we propose a comprehensive machine learning pipeline designed to enhance fraud detection accuracy by integrating advanced feature engineering, outlier analysis, and comparative model evaluation. We begin by transforming raw transactional data into informative features, such as account balance behavior (emptied_account), temporal transaction indicators (hour_risk_level), and custom behavioral flags (e.g., matching amount and balance). These engineered features are supplemented with outlier indicator variables that capture extreme financial anomalies, enabling the model to retain important variance while mitigating noise through winsorization. To evaluate the effectiveness of our feature-enriched dataset, we train and compare several state-of-the-art classification models: CatBoost, XGBoost, LightGBM, Random Forest, K-Nearest Neighbors (KNN), and Logistic Regression. All models are evaluated using a balanced set of performance metrics including accuracy, precision, recall, F1-score, and the Area Under the Receiver Operating Characteristic Curve (ROC-AUC). Additionally, we address the issue of class imbalance by applying cost-sensitive learning through class weighting, particularly in boosting models like CatBoost. The performance is further scrutinized through overfitting analysis, using both training and test metrics to quantify generalization. Our experiments show that gradient boosting models, particularly CatBoost, consistently outperform traditional models in detecting fraudulent transactions, thanks to their ability to handle categorical variables, class imbalance, and complex feature interactions effectively. The study emphasizes the critical role of domain-specific feature engineering and cost-aware learning strategies in developing robust fraud detection systems. These findings provide a scalable and interpretable framework for financial institutions seeking to enhance their fraud prevention capabilities in real time.