GAShellBreaker: A Novel Method for Java Fileless Webshell Detection Based on Grayscale Images and Deep Learning

Webshells are widely used by attackers to maintain access during the post-exploitation phase. As security defenses improve, traditional file-based Webshells are increasingly detectable. To evade detection, attackers are shifting toward fileless Webshells, which reside entirely in memory and present significant challenges to conventional security tools. However, research on fileless Webshell detection remains limited. To address this gap, we analyzed various fileless Webshell samples, summarized their behavioral patterns, and constructed a corresponding threat model. Based on this, we propose a novel detection approach named GAShellBreaker, which leverages grayscale image transformation and deep learning. GAShellBreaker first establishes a dual-layer in-memory monitoring mechanism to capture suspicious classes within the Java Virtual Machine (JVM) and export them as bytecode files. It then extracts opcode sequences from these files, transforms them into grayscale images, and employs a ResNet50-based classifier for detection. Due to the limited availability of fileless samples, we trained and evaluated the model on a larger dataset of 1351 file-based scripts (383 Webshells and 968 benign samples), and used 56 fileless Webshells for validation. Experimental results show that GAShellBreaker achieves 99.10% accuracy on file-based Webshells and 89.29% accuracy on fileless Webshells, outperforming existing algorithms. Moreover, it maintains low computational overhead (6.7%), confirming its practical feasibility.