SHAP-guided feature refinement for efficient, robust, and interpretable memory-forensic malware detection

The evolution of malware continues to provide a challenge to existing detection methods, particularly those based on static analysis and signature-based heuristics. This paper investigates a memory-forensics-driven methodology for malware detection that includes feature refinement, model explainability, efficiency analysis, adversarial robustness testing, and family-disjoint validation. Using the CIC MalMem-2022 dataset, we examined 55 memory-resident features and reduced them to 13 using mutual information and SHapley Additive exPlanations-guided feature refinement (SHAP-GFR). We tested five classifiers: RF, XGBoost, (RF), eXtreme Gradient Boosting (XGBoost), multilayer perceptron (MLP), one-dimensional CNN (1D CNN), and a CNN-LSTM hybrid using a 70% train, 15% validation, and 15% test split. On clean independent and identically distributed (IID) data, RF had the greatest F1-score of 0.9999, while XGBoost had a comparable F1-score of 0.9997, the lowest median per-sample inference latency of 0.0054 ms, and the smallest model size of 0.312 MB. Using SHAP and Local Interpretable Model-agnostic Explanations (LIME), we discovered that service and handle-related memory features had the greatest impact on model decisions, followed by module-loading and injection-related features. Our robustness tests using Gaussian noise, Fast Gradient Sign Method (FGSM), Momentum Iterative Fast Gradient Sign Method (MI-FGSM), Projected Gradient Descent (PGD), and surrogate transfer attacks revealed that model resistance was affected by both architecture and attack type. We found that CNN-LSTM and 1D CNN were more robust under direct gradient-based perturbations, while tree ensembles were more vulnerable to Gaussian and transfer perturbations. Parsed-family Leave-One-Family-Out (LOFO) validation revealed significant cross-family generalization for RF and XGBoost, with mean F1-scores of 0.9988 and 0.9987, respectively, but large families remained the most difficult. These results render the clean-accuracy tale incomplete: a model that performs well on IID data may yet differ drastically in latency, interpretability, robustness, and behavior on unidentified malware families.