Binary protocol greybox fuzzing driven by marine predators algorithm

Abstract Binary protocols are widely used in the Internet, industrial control systems, and Internet of Things applications, and vulnerabilities in their implementation can directly affect system security. Existing binary-protocol fuzzers face three practical challenges: identifying effective mutation positions in messages with implicit field boundaries, inferring mutation constraints for different protocol fields, and scheduling appropriate mutation operators to explore new program paths without generating excessive invalid test cases. This paper proposes MPAuzz, a greybox fuzzer for binary protocols driven by the Marine Predators Algorithm (MPA). MPAuzz first performs feedback-based mutation position exploration and classifies message regions as mutable, restricted, or immutable. It then formulates mutation operator scheduling as a multidimensional optimization problem, where each dimension corresponds to an operator choice for a mutation region. By simulating the staged exploration and exploitation process of MPA, MPAuzz adaptively adjusts operator combinations according to program responses and coverage feedback. Experiments on binary protocol implementations including MQTT, DTLS, DNS, and CoAP show that MPAuzz achieves an average valid-test-case ratio of 98.1%, improves branch coverage by 38.3% over AFLNet and 26.5% over StateAFL on average, and triggers the highest number of crashes across all targets.