MemHookNet: Real-Time Multi-Class Heap Anomaly Detection with Log Hooking

Heap memory anomalies, such as Use-After-Free (UAF), Double-Free, and Memory Leaks, pose critical security threats including system crashes, data leakage, and remote exploits. Existing methods often fail to handle multiple anomaly types and meet real-time detection demands. To address these challenges, this paper proposes MemHookNet, a real-time multi-class heap anomaly detection framework that combines log hooking with deep learning. Without modifying source code, MemHookNet non-intrusively captures memory operation logs at runtime and transforms them into structured sequences encoding operation types, pointer identifiers, thread context, memory sizes, and temporal intervals. A sliding-window Long Short-Term Memory (LSTM) module efficiently filters out suspicious segments, which are then transformed into pointer access graphs for classification using a GATv2-based model. Experimental results demonstrate that MemHookNet achieves 82.2% accuracy and 81.5% recall with an average inference time of 15 ms, outperforming DeepLog and GLAD-PAW by 11.7% in accuracy and reducing latency by over 80%.