AFLDL: state-aware greybox protocol fuzzing based on deep learning

Abstract Fuzzing stateful security protocols presents a significant technical challenge. The behavior of the target server depends not only on the content of individual messages but also on the evolution of its internal state across message sequences. This statefulness imposes severe limitations on traditional code-coverage-guided fuzzing techniques. As a result, effective fuzzing of stateful protocols requires state-aware strategies, which may originate from manual guidance, automated analysis, or a combination of both. Automated analysis methods typically assume that researchers lack prior knowledge of the protocol and rely on heuristic approaches to infer a protocol state machine during interactions, which is then used to guide the fuzzing process. However, in more common practical scenarios, researchers often have partial knowledge of the protocol specification or implementation, enabling them to manually construct parts of the state machine. When conflicts arise between automatically inferred states and expert-defined ones, current approaches lack effective mechanisms to integrate human prior knowledge into the state machine refinement process. To address the above challenges, this paper proposes AFLDL, a state-aware grey-box fuzzing approach based on deep learning. AFLDL leverages neural networks to establish mappings between memory snapshots and protocol states derived from prior knowledge. During fuzzing, it infers the current protocol state of the target system by analyzing runtime memory snapshots, and discovers new states through clustering analysis. These inferred states are used to progressively build a complete state machine representation that guides state-aware fuzzing. Experimental results demonstrate that AFLDL outperforms state-of-the-art fuzzing tools, achieving higher code coverage and triggering more crashes within the same time frame.