SETS: A Simple yet Effective DNN Test Selection Approach

To reduce the substantial manual annotation costs in testing deep neural networks (DNNs), various test selection approaches have been proposed, with uncertainty serving as a key objective to trigger more failures. However, focusing solely on uncertainty may expose failures caused by the same underlying fault in the model. Accordingly, recent studies have introduced diversity as another important objective, proposing multi-objective search-based approaches to select test inputs that can reveal distinct faults. Despite their effectiveness, these approaches typically require long execution times, limiting their efficiency in large-scale datasets. In this paper, we propose SETS, a simple yet effective multi-objective DNN test selection approach. Like existing multi-objective approaches, SETS optimizes both uncertainty and diversity to maximize the number of unique faults detected by the selected test inputs. But, SETS prioritizes high-uncertainty test inputs to reduce the candidate set size, and employs an efficient greedy strategy to further reduce the number of fitness evaluations. Thus, SETS differentiates itself by considerably improving the efficiency of selection while preserving effectiveness. We evaluate SETS on eight subjects, demonstrating its significant superiority over baseline approaches. SETS can operate on average 42.98 times faster while still achieving better fault detection and model retraining with its simple implementation.