Advancements in Machine Learning and Deep Learning for Malware Detection Challenges Breakthroughs

Malware detection remains a critical concern in cybersecurity, exacerbated by the increasing sophistication of cyber threats and the inadequacies of conventional signature-based approaches. This study aims to investigate recent advancements in Machine Learning (ML) and Deep Learning (DL) techniques that address the evolving challenges in malware detection. Focusing on hybrid architectures, ensemble methods, and the integration of Explainable Artificial Intelligence (XAI), this systematic review analyzes the effectiveness of state-of-the-art models trained on widely used public datasets, including Malimg, CIC-MalMem-2022, and the Windows PE Malware Dataset. The review identifies key limitations such as adversarial susceptibility, dataset dependency, and limited real-time applicability, which hinder deployment in practical environments. By critically examining the influence of dataset diversity on model robustness and generalization, the study highlights that while many models demonstrate high accuracy in controlled settings, their performance often deteriorates in dynamic, real-world scenarios. The findings underscore the necessity of advancing toward more resilient and interpretable detection frameworks. As a contribution to the field, the review proposes a future research agenda centered on the development of adaptive hybrid models, adversarial defense mechanisms, and standardized real-world benchmark datasets. These directions are essential for enhancing the reliability, scalability, and transparency of next-generation malware detection systems.