Real-Time Detection of Windows Shellcode Injection via the Cross-Process W^X Principle

Abstract Shellcode injection allows attackers to execute arbitrary code inside a target process without writing files to disk, and remains a common technique in modern malware and advanced persistent threats. Existing defenses cover only part of this threat landscape: Data Execution Prevention (DEP) prevents a process from executing its own data pages, and Arbitrary Code Guard (ACG) restricts dynamic code operations initiated by protected processes. However, neither mitigation prevents a remote process from acquiring executable memory in a target process through cross-process system calls. This paper presents XMemGuard , a kernel-mode detection framework that enforces the cross-process write-XOR-execute ( $$\text {W}\,\hat{\phantom{a}}\,\text {X}$$ W a ^ X ) principle: cross-process system calls must not create attacker-controlled executable memory in the target process. XMemGuard employs a pre-dispatch system-call interception mechanism based on Windows Event Tracing (ETW). This mechanism intercepts three categories of system calls responsible for cross-process acquisition of executable memory, removes the execution bit during the acquisition stage, and then uses an in-process monitoring component to capture the runtime outcomes of subsequent execution attempts. Compared to three existing detection methods, XMemGuard can detect all 33 manually constructed PoC variants, while the strongest baseline method can only detect 30. On in-the-wild samples, it achieves the highest accuracy, recall, and F1 score among the compared methods, with 99.79% precision. Furthermore, performance evaluations show that the overhead of XMemGuard is mainly concentrated in workloads with intensive cross-process memory operations, with a limited impact on overall system performance.