Characterizing Regression in Deep Learning Model Evolution

Deep learning (DL) models have seen extensive application across diverse fields. As these models continue to advance for enhanced predictive capabilities, instances of performance degradation, or regression, have emerged. Yet, there’s limited understanding of how this regression manifests across different datasets and models. To address this gap, we conduct an extensive empirical analysis using four datasets, three deep neural network (DNN) models, seven comtempory methods for selecting test inputs, and four advanced techniques aimed at mitigating regression. Our key findings include:(1)Regression is a common occurrence during the evolution of DL models across various datasets and architectures, often overshadowing gains in accuracy. (2)Test input selection strategies based on uncertainty can enhance accuracy but may introduce additional regression issues.(3)Current techniques designed to reduce regression are not consistently effective and typically involve a trade-off between mitigating regression and improving overall performance. These insights offer practical guidance for the selection of test inputs and the management of regression in the ongoing development of DL models.