RLGFuzz: Reinforcement Learning Guided Fuzzing with State-Coverage Mapping Environment

To ensure the security and reliability of protocol implementations, protocol fuzzing serves as a crucial technique of security assessment and is widely adopted. However, existing fuzzers exhibit low efficiency after a certain duration of testing, primarily due to the lack of more flexible and adaptive guidance in seeds and states selection. In response to this issue, we propose a reinforcement learning guided fuzzer named RLGFuzz. Neural networks are employed to learn potential mappings between seeds and states to path coverage, which are then utilized in an interactive environment for reinforcement learning. This process allows the fuzzer to preferentially test seeds and states. Experimental results on a diverse set of real-world protocols demonstrate significant improvements in path coverage and triggering crashes efficiency achieved by our RLGFuzz. Compared to AFLNet, AFLNwe, and SMGFuzz, RLGFuzz outperforms all three, achieving a 6.68% enhancement in path coverage and a 5.52% enhancement in efficiency of triggering crashes.