An Overview of Adaptive Dynamic Deep Neural Networks via Slimmable and Gated Architectures

Deep Neural Networks (DNN) are omnipresent in systems that are developed for processing vast quantities of data, most particularly images, for tasks such as perception, navigation, classification, detection, and segmentation. Such DNNs can be computationally demanding from the high number, upwards to hundreds of billions, of computations required during execution. Dynamic Deep Neural Networks (DDNN) are an evolution that allow the number of computations in a DNN to be scaled down. However, lacking in DDNN methodologies is the functionality to control online when and how to scale down the number of computations. To respond to this need, Adaptive Dynamic Deep Neural Networks (ADDNN) are an evolving class of deep learning models used in high performance computing that attempt to minimize resources usage - memory and power - and latency while maintaining an acceptable task performance by adapting the model architecture in response to the current context. Thus, ADDNNs are a State-of-the-Art evolution that perceive context, on a case-by-case basis, and adapt the number of computations to that required by the difficulty of the problem at hand. This positional paper is not only a survey of current DDNN methods in literature, but also an analysis into the considerations, design principles and challenges in developing robust ADDNN systems and applications.