Hybrid Machine Learning for Enhanced Ransomware Detection: A Random Forest and XGBoost Approach

Ransomware attacks continue to threaten global cybersecurity, crippling systems through data encryption and extortion. Traditional detection methods struggle with evolving threats like zero-day and polymorphic ransomware. This paper introduces a hybrid machine learning framework that combines Random Forest for anomaly detection and Extreme Gradient Boosting(XGBoost) for classification, addressing critical gaps in accuracy and false alarms. By analyzing structural and behavioral features-such as DebugSize, IatVRA, and ResourceSize our model identifies ransomware with exceptional precision. Trained on a real-world dataset of 12,497 samples (7,073 benign, 5,424 malicious), the hybrid approach achieves 99.52% accuracy, 99.47% precision, and 99.43% recall, outperforming standalone XGBoost (99.34% accuracy) and Random Forest (99.41%) accuracy. Our hybrid model achieves a 0.41% false positive rate, a 49–99% reduction over Decision Tree, SVM, and Naive Bayes. SHapley Additive exPlanations analysis highlights DebugSize and irregular import address table behavior as critical ransomware indicators, enhancing model transparency. These results demonstrate a robust, interpretable solution for real-world deployment, offering cybersecurity professionals a reliable tool to combat sophisticated attacks.