FPGA Interactive Debugging Tools Testing via Mutation Diversification Search

FPGA interactive debugger tools assist engineers in quickly detecting and correcting bugs in RTL designs through powerful built-in interactive debugging tools. The interactive debugging process ensures the accuracy and development efficiency of RTL designs. Rigorous testing of such FPGA interactive debugging tools is crucial, as engineers may misinterpret RTL designs and introduce incorrect fixes due to bugs in FPGA interactive debugging tools, leading to security risks. In this study, we propose a novel differential testing method called ID-Hunter, which is the first study to leverage the mutation diversification search guided via the Multi-Armed Bandit (MAB) for generating RTL designs to test FPGA interactive debugging tools. ID-Hunter consists of three components: the mutation rule component, the interactive mutation optimization component, and the differential testing component. The mutation rule component designs two types of mutation rules (i.e., program transformation and interactive debugger action transformation) to generate equivalent RTL designs during interactive debugging. Then, the combined mutation optimization component conducts mutation diversification search via MAB for guiding the construction of effective RTL designs based on the memoization for the explored mutation rules during the on-the-fly interactive debugging process. ID-Hunter then detects bugs by analyzing the outputs of these RTL designs by the differential testing component. Specifically, ID-Hunter considers the diversity among mutation rules to efficiently explore the input space and the testing results under each explored mutation rules to learn which portions of space are more bug-triggering. Within three months, ID-Hunter reported 15 bug issues, 10 of which were confirmed by Xilinx support as bugs, with 3 of them were fixed.