Self-adaptive fuzzing optimization method based on distribution divergence

To improve the performance of coverage-guided fuzzing, a method for self-adaptive optimization of fuzzing using distribution divergence and a deep reinforcement learning model was proposed. An interprocedural comparison flow graph was first constructed based on the interprocedural control flow graph to characterize the spatial random field corresponding to the branch condition variables of the program under test, and the distribution features of the random field generated by a fuzzing mutation strategy were extracted using the Monte Carlo method. Then, a deep graph convolutional neural network was constructed to extract the feature embeddings of the interprocedural comparison flow graph, and this neural network was used as the deep Q-network for deep reinforcement learning. Finally, an online deep reinforcement learning model was established based on the dual deep Q-network model, and an intelligent agent was trained to optimize the fuzzing mutation strategy. This deep reinforcement learning model defined its state using the random field distribution features corresponding to the seed file and the associated blocks. The selection for the focused mutation block of a seed file was defined as an action, and the distribution divergence of the approximate distributions of the random fields before and after the action was defined as the reward. A prototype was implemented for this fuzzing optimization method, and multiple rounds of up to 24 hours of evaluation were carried out on this prototype. The experimental results show that on the benchmark FuzzBench, the code coverage speed and overall coverage achieved by the prototype are significantly better than those of the baseline fuzzer AFL++ and HavocMAB, and better results are achieved on most benchmarks compared to CmpLog. On the benchmark Magma, stronger vulnerability triggering capability is demonstrated by the prototype on the benchmarks openssl, libxml, and sqlite3.