Adaptive Detection of Fileless & Polymorphic Malware Using Memory Forensics, Computer Vision, and Machine Learning

The rise of fileless and polymorphic malware presents significant challenges for traditional detection methods, as these sophisticated threats operate exclusively in memory or continuously modify their code to evade signature-based detection. This paper proposes an enhanced detection framework that leverages memory forensics, computer vision, and machine learning to identify these evasive malware types. Our approach converts memory dumps into RGB images and applies a combination of GIST and HOG descriptors to extract distinctive visual features, which are then classified using various machine learning algorithms. Experimental results on a dataset of 3953 samples across 11 classes (10 malware families and benign executables) demonstrate superior classification accuracy of 97.49% using Linear SVM. Furthermore, we employ the Uniform Manifold Approximation and Projection (UMAP) technique to improve the detection of unknown malware, achieving accuracy improvements of up to 36.98% in binary classification tasks. The proposed methodology offers a robust solution for detecting advanced mal-ware that bypasses conventional security measures, with minimal false positives and real-time detection capabilities. This research bridges the fields of memory forensics and computer vision, establishing a novel approach to cybersecurity that addresses the increasing sophistication of modern malware threats.