Evaluation on parameter-efficient continual instruction tuning of large language models

Recent years have witnessed a spurt in large language models for its strong generalization performance, many continual instruction tuning methods based on parameter-efficient tuning have been proposed to further push large language models towards artificial general intelligence. However, comprehensive assessments of recent advancements in these methods are notably lacking. To fill the gap, a three-dimensional (average performance, continual learning proficiency and general ability delta) evaluation protocol are introduced based on two task streams, which include wide-span tasks and long sequence tasks. Meanwhile, three continual instruction tuning methods following different strategies are thoroughly evaluated on three distinct language models. The empirical analysis reveals that regularization-based methods are well-suited for wide-range task streams, while replay-based methods excel in long sequence task streams, particularly for tasks of rich logic reasoning in maintaining general ability. Simultaneously, the study underscores the need for new continual instruction tuning methods based on parameter-efficient tuning that balance performance on new tasks with the preservation of general capabilities, especially for more sophisticated architecture-based method.