MetBench: A Numerical Expression Metamorphic Relations Benchmark Data Set

Metamorphic testing is one of the most effective methods to alleviate the oracle problem. However, the lack of benchmark data sets has hindered the development of metamorphic testing. This study identifies and extracts key characteristics from three entities: the business domain, applications, and metamorphic relations. It then establishes a model for numerical expression metamorphic relations. Data were gathered from open source programs, academic papers, and industrial software, which resulted in the construction of the MetBench benchmark data set encompassing 139 codes and 924 metamorphic relations. In addition, we developed a general purpose tool for automating the execution of metamorphic testing, which separates code, metamorphic relations, and testing processes. This tool supports the management of metamorphic relations, the automated execution of metamorphic testing, and the visualization of the results. Specifically, metamorphic relations are encoded in the literal format of LaTeX and the computational format of SymPy and are stored using the document-oriented database LiteDB. The system utilizes a combination of C# and Python programming languages. The former is responsible for the user interface, while the latter handles the automated execution. Visualization is achieved through Matplotlib. The establishment of the MetBench benchmark data set facilitates the study of metamorphic relation identification techniques, the evaluation of the effectiveness of relations, and their nature characteristics such as complexity, similarity, and false satisfaction rate.