Dual-use ambiguity in static ransomware detection: false positive analysis and confidence-gated mitigation

Static machine learning models are widely used for ransomware detection because Portable Executable (PE) features can be extracted before execution and often produce strong benchmark results. However, many PE-level indicators, including entropy, imports, API calls, resource metadata, and section-level attributes, are also present in legitimate encryption and compression utilities. This overlap creates a dual-use ambiguity that conventional benign–malicious benchmark testing may not reveal. This study evaluates five classifiers—Random Forest, XGBoost, LightGBM, Logistic Regression, and Support Vector Machine—using a 2,675-sample ransomware benchmark dataset and a separate 300-sample dual-use stress test containing benign encryption, compression, backup, password-management, cryptographic, secure-deletion, and file-protection tools. To eliminate data leakage, models are evaluated under a strict family-disjoint protocol across 40 ransomware family clusters. The results of the study suggest that under the family-disjoint evaluation, standalone XGBoost achieved 88.69% ransomware recall on previously unseen families but produced a 57.89% false-positive rate on the dual-use holdout, demonstrating that strong ransomware generalization does not necessarily translate into reliable discrimination of legitimate encryption-related software. The study then proceeds to develop a confidence-gated two-step approach, which employs an XGBoost first-stage classifier and an SVM second-stage classifier evaluated on a version-disjoint 209-sample dual-use holdout. The results demonstrate that the proposed approach decreases the dual-use false-positive rate to 35.41% on unseen ransomware families while maintaining 59.36% ransomware recall, outperforming the baseline of a standalone SVM with 37.32% false-positive rate. The results also illustrate the trade-off between the false-positive rates and ransomware recall of the confidence-thresholding approach compared to training a classifier on the test set directly and decision threshold optimization with score averaging. The score-averaging ensemble achieved a ransomware recall of 83.39% with a dual-use false-positive rate of 45.93%, while decision-threshold optimization for F1 score retained a higher ransomware recall of 86.22% but resulted in a 49.28% false-positive rate. These findings demonstrate that neither baseline adequately resolves the trade-off between ransomware detection and false-positive reduction, motivating the use of the proposed two-stage confidence-gated classification framework for more controlled handling of ambiguous dual-use samples.