Is Fault Localization Effective on Industrial Software? A Case Study on Computer-Aided Engineering Projects

In software engineering, empirical studies on automated fault localization (FL) methods mainly focus on general software, and substantial progress has been made. However, the applicability and efficacy of these methods in specialized, domain-specific software like industrial software remains under-explored. Such specialized software is usually characterized by complex inputs and iterative computing paradigms, which could significantly influence the effectiveness of existing FL methods. To address this gap, this study takes a typical categorical of industrial software (i.e., computer-aided engineering (CAE) projects) as a case study, to investigate the feasibility and effectiveness of state-of-the-art FL methods within CAE projects. Through the reproduction of 76 real-world bugs from three widely used CAE projects (i.e., FDS, deal.II, and MFEM), we find that even the most precise FL methods require developers to examine on average 467.18 statements before finding bugs and can take 208.13 hours to execute. The complex inputs and long-term computation characteristics of CAE projects further increase the difficulty of FL. Moreover, FL on CAE also faces challenges, such as insufficient differentiation of coverage information and missing CAE-specific FL features. Based on our findings, we improve FL on CAE projects by proposing a set of CAE main module-based features, which improve the best-performed FL method in this study (i.e., DeepFL) by 35.93% and 45%, in terms of MAR and MFR , respectively.