Revisiting Diversity Metrics and Coverage Criteria for Deep Neural Networks Quality Assessment from the Perspective of Test Adequacy

Deep neural networks (DNN) have extensive applications in image processing, medical diagnosis, autonomous driving, and various other domains. Previous research has found limitations in the error-awareness capacities of deep neural networks, especially when deployed in safety-critical systems. Several neuron coverage criteria have been suggested to direct the adequate testing and quality assessment of deep neural networks, drawing inspiration from conventional software quality assessment methodologies. Nevertheless, recent research has raised doubts about the validity of these criteria in guiding quality assessment of deep neural networks and measuring test adequacy. In this paper, we use ImageNet-A as an additional test dataset and employ three diversity metrics to assess the diversity of the hybrid datasets comprising both ImageNet and ImageNet-A. Subsequently, we employ the hybrid dataset in conjunction with eight DNNs, to investigate the statistical correlation between the three diversity metrics and the DNNs’ accuracy in predicting correct outcomes. Furthermore, we conduct a comparative analysis between the diversity metrics and DNN accuracy, as well as between coverage criteria and DNN accuracy. The classification task used in the experiment is also widely used in various sensors for ubiquitous intelligence. The results of our experiment indicate that test set diversity metrics serve as superior indicators of test adequacy compared to coverage criteria, thereby enabling more effective guidance for testing DNNs.