Why and How We Combine Multiple Deep Learning Models With Functional Overlaps

ABSTRACT The evolution (e.g., development and maintenance) of deep learning (DL) models has attracted much attention. One of the main challenges during the development and maintenance of DL models is model training, which often requires a lot of human resources and computing power (such as labeling costs and parameter training). In recent years, to alleviate this problem, researchers have introduced the idea of software engineering (SE) into DL. Researchers consider the DL model a new type of software, borrowing the practice of traditional software reuse, that is, focusing on the reuse of DL models to improve the quality of DL model development and maintenance. This paper focuses on more complex model reuse scenarios, where developers need to combine multiple models with functional overlaps. We explore whether the model combination technique can meet the requirements for such scenarios. We have conducted an empirical study of the research scenario and found that a model composition approach was needed to meet the requirements. Furthermore, we propose a model combination method based on concatenation‐parallel called MCCP. First, the multiple models' hidden layer features are connected, and then the multiple models are connected in parallel to construct a joint model with all output categories. The joint model is trained to achieve unified requirements under the limited marking cost. Through experiments on data sets in nine domains and five model structures, the following two conclusions are drawn: (1) we observe noticeable differences (38% at most) in the performance of multiple models within overlapping category data, which calls for effective model combination techniques. (2) MCCP is more effective than the baseline, which performs the best in eight of the nine domains. Our research shows that the joint model generated by combining models with overlapping functions can meet the requirements of complex model reuse scenarios.