Composing Lightweight Methods on Deep Neural Network: A Survey

Abstract Deep learning has achieved remarkable progress across many domains, largely powered by large-scale neural networks. Their substantial computational demands, however, hinder deployment on resource-constrained edge devices. To address this, lightweight techniques, such as efficient architecture design, pruning, quantization, decomposition, and knowledge distillation, have been explored to reduce the required resources while preserving accuracy. While prior surveys typically investigate these methods one by one, limited attention has been given to how their composition, ordering, and optimization dependencies affect efficiency and accuracy in real-world settings. In this survey, we introduce a taxonomy of lightweight method compositions, organized by which lightweight methods are selected and how those methods are composed and interact, including sequential application, joint optimization, and hybrid approaches. We qualitatively and quantitatively analyze these categories, with particular attention to within-study comparisons, to assess the effectiveness of different compositions in improving efficiency and accuracy, providing guidance for future research in this area and practical deployment of deep neural networks.