Optimizing Classification of Suspicious Financial Transactions Using Machine Learning: a Comparative Study of Random Forest, Xgboost, and Svm

This study explores the application of machine learning models to detect suspicious financial transactions in the banking sector. Using a dataset of 12,571 transactions obtained from PT Bank ABC, the research follows a comprehensive methodology involving data preprocessing, feature selection, and handling class imbalance. Three models—Random Forest, XGBoost, and Support Vector Machine (SVM)—were implemented and evaluated using Stratified K-Fold crossvalidation. To enhance model performance, hyperparameter tuning was performed using RandomizedSearchCV. The Random Forest model achieved the highest performance, with an accuracy of 99.52%, precision of 98.23%, and recall of 99.11%. XGBoost followed closely with 99.49% accuracy, 97.72% precision, and 99.46% recall. The SVM model also demonstrated solid results, achieving 95.20% accuracy, 89.94% precision, and 86.67% recall. These outcomes indicate that all three models are capable of identifying suspicious transactions, with Random Forest and XGBoost proving particularly effective in managing complex and imbalanced data. The findings highlight the potential of machine learning to strengthen financial security systems by improving the early detection of fraudulent activity. This research contributes to the growing body of evidence supporting the use of intelligent systems for risk management in the banking industry.