Neural networks-based automated test oracles

Abstract With increasing complexity and importance in the society of software systems, testing software products becomes increasingly challenging and one of the problems that appears is determining the correct output given an input, named the test oracle problem. The aim of the paper is two fold: (1) to replicate the findings in a previous work regarding the use of various neural network models for the test oracle problem, and (2) to investigate further aspects regarding hyperparameter optimization by using the L4 Taguchi approach. Three datasets were used, two from previous studies and one created. In the execution of the experiment it is also simulated the real regression testing process by using mutation datasets. The results from the replication experiments show similar results to the ones from the original research, on the Triangle dataset, the ANN (Artificial Neural Networks) has a 0.12 MARE (Mean Absolute Relative Error) score while RBF (Radial Basis Function) 0.23; however, for both Bank Credit and Heart Risk datasets the RBF obtained the best MARE results, 0.09 and 0.12. The Taguchi L4 method does not offer a single optimal solution for a model for all datasets, but we can observe some trends in the results. Across all experiments, for the epochs parameters it seems that the best results are for 100 for all datasets and for the two models, except for the triangle dataset and ANN model. Regarding the hidden layers, best results are for the 50 nodes in the case of the ANN and 350 nodes in case of the RBF.