Fileless Malware Threats: Evaluating Machine Learning Algorithms for Anomaly Detection

Fileless malware is an advanced cybersecurity threat that utilizes trusted system modules to execute itself without leaving traces detectable by traditional antivirus software. Detecting this memory-resident malware requires an anomaly detection approach. This study evaluates the performance of four different unsupervised learning techniques: Autoencoder, One-class Support Vector Machine (SVM), Isolation Forest, and Local Outlier Factor (LOF) regarding fileless malware detection. The experiment revealed that LOF achieved the highest accuracy at 87.06%, while One-class SVM was the least efficient. Ultimately, this article suggests that future research should aim to reduce false negatives and adopt hybrid techniques to improve detection against evolving cyber threats.