Enhancing Linux System Security: A Kernel-Based Approach to Fileless Malware Detection and Mitigation

In the late 20th century, computer viruses emerged as powerful malware that resides permanently in target hosts. For a virus to function, it must load into memory from persistent storage, such as a file on a hard drive. Due to the significant destructive potential of viruses, numerous defense measures have been developed to protect computer systems. Among these, antivirus software is one of the most recognized and widely used. Typically, antivirus solutions rely on static analysis (signature-based) technologies to detect infections in files stored on permanent storage devices, such as hard drives or USB (Universal Serial Bus) flash drives. However, a new breed of malware, fileless malware, has been designed to evade detection and enhance durability. Fileless malware resides solely in the memory of the target hosts, circumventing traditional antivirus software, which cannot access or analyze processes executed directly from memory. This study proposes the Check-on-Execution (CoE) kernel-based approach to detect fileless malware on Linux systems. CoE intervenes by suspending code execution before a program executes code from a process’s writable and executable memory area. To prevent the execution of fileless malware, CoE extracts the code from memory, packages it with an ELF (Executable and Linkable Format) header to create an ELF file, and uses VirusTotal for analysis. Experimental results demonstrate that CoE significantly enhances a Linux system’s ability to defend against fileless malware. Additionally, CoE effectively protects against shell code injection attacks, including buffer and memory overflows, and can handle packed malware. However, it is important to note that this study focuses exclusively on fileless malware, and further research is needed to address other types of malware.