Towards Scalable Evaluation of Software Understanding: A Methodology Proposal

In reverse engineering our goal is to build systems that help people to understand software. However, the field has not converged on a way to measure software understanding. In this paper, we make the case that understanding should be measured via performance on understanding-questions. We propose a method for constructing understanding-questions and evaluating answers at scale. We conduct a case study in which we apply our method and compare Ghidra’s default auto analysis with an analysis that supports binary constructs that are specific to Objective-C.