Fuzz Testing of Protocols Based on Protocol Process State Machines

Network protocols are important components of network communication systems, supporting interactions and communications among different entities in the network. Their security deserves increased attention and research. Fuzz testing, known for its high efficiency and low false positive rates, has been widely applied in modern vulnerability discovery efforts, yielding significant results in the field of software security. Existing fuzz testing methods primarily focus on local applications, with relatively few studies dedicated to fuzz testing for network protocols. Current fuzzers have several limitations, such as inaccurate state representation, the inability to effectively target critical states for testing, and a lack of effective mutation strategies. To address these issues, this paper presents a protocol fuzz testing method based on protocol process state machines. This approach utilizes state variables identified within the source code to obtain additional process states, establishing a mapping between the state space and the code space to construct a protocol process state machine. Additionally, it involves assigning weights to each state to implement a state selection algorithm that focuses on targeted fuzz testing of critical states. Experimental results from fuzz testing on protocol service programs such as TinyDTLS and ProFTPD show that the tool ZFuzz achieves up to a 30.97% improvement in edge coverage compared to AFLNet and up to a 10.93% improvement compared to NSFuzz. ZFuzz can cover all crashes triggered by the comparative tools and discover 2–4 additional crashes that were not identified by the baseline programs. Moreover, the average time to the first crash trigger is enhanced by 78.10%. These findings suggest that this tool has superior coverage capabilities and vulnerability discovery efficacy.