Hybrid AI-Based Static Analysis for Malware Detection: A Feature Engineering and Model Optimization Approach

In this study, we propose an optimized ensemble-based static malware detection framework that emphasizes feature selection, hyperparameter tuning, and robustness to imbalanced datasets. Unlike prior works that focus primarily on interpretability or balanced classification scenarios, our approach addresses practical deployment challenges in real-world environments where benign files are significantly outnumbered by malicious ones.Using a dataset of 11,000 Windows PE files (10,000 malware, 1,000 benign), we extract a diverse set of metadata-based features including PE structure details, byte-level statistics, and imported libraries. We apply SHAP (SHapley Additive Explanations) not only to explain model decisions but to perform feature reduction by eliminating low-impact attributes, enhancing training efficiency.To improve model generalization, we conduct rigorous hyperparameter optimization using Optuna and GridSearchCV across multiple classifiers, including LightGBM, XGBoost, Random Forest, and MLP. LightGBM achieved a peak accuracy of 97.23%, and the stacking ensemble yielded stable and comparable results.Additionally, we discuss the limitations of static-only analysis and outline plans for incorporating adversarial robustness and dynamic behavior features in future iterations. The findings demonstrate that tuning model complexity and feature space can lead to highly accurate, interpretable, and deployable malware detection systems.