What Causes Bugs in Numerical Simulation Software? An Empirical Study

Numerical simulation (NS) software is widely used in safety-critical domains (e.g., aerospace design) to simulate actual physical processes of real-world entities on computers. However, NS software is error-prone, whose bugs lead to incorrect simulation, and may even cause disastrous flaws in safety-critical applications. Although many studies investigate the bug characteristics of computation-centered software, such as machine learning systems, the characteristics of NS software bugs have not been fully studied: what are the root causes and symptoms; how are they different from other computation-centered software; and why the difference occurs. To bridge this gap, we present a systematic study of NS software bugs by analyzing 352 bugs in three popular NS projects (i.e., FDS, SU2, and Kratos) for different domains. We summarize seven root causes (with 18 subcategories) and five symptoms. We find that the correctness, completeness, compatibility, and parallelization to implement NS algorithms (i.e., models) are error-prone. Many root causes (e.g., incorrect model and incorrect initialization) require physical and chemical knowledge to avoid bugs, which may not be mastered by typical software developers. These findings motivate new challenges and opportunities for future NS software development, such as designing domain specific language systems for NS model.