From full fine-tuning to parameter-efficient adaptation: a taxonomy of deep transfer learning strategies

This paper focuses on the field of deep transfer learning, aiming to study the dual dilemma of traditional full-parameter fine-tuning in terms computing power demand and overfitting risk. By establishing a three-level taxonomy of parameter-efficient transfer learning, this paper analyzes in detail the methods such as parameter addition strategy parameter reparameterization, and parameter subset selection. It also refines the general principles such as minimal disturbance of feature space, task-knowledge decoupling, and incremental knowledge injection Simultaneously, it points out the current challenges of lack of theoretical interpretability (black-box nature) and weak cross-architecture generalization ability, and proposes future directions such as-explanation-based interpretable adaption and neural architecture search (NAS)-driven automatic adaption, aiming to construct a more systematic and efficient theoretical framework for deep transfer learning.