MT2ST: Adaptive Multi-Task to Single-Task Learning

We propose MT2ST, a general and efficient framework for accelerating multi-task training by progressively transitioning to single-task optimization.Unlike conventional multi-task learning (MTL) or singletask fine-tuning (STL), MT2ST dynamically adjusts the training focus via two complementary strategies: Diminish, which gradually down-weights auxiliary losses, and Switch, which explicitly switches to the primary task at a scheduled point.We demonstrate the effectiveness of MT2ST across three key paradigms: representation learning, transformers, and diffusion models, covering both unimodal (text/image) and multimodal (vision-language) tasks.Extensive experiments show that MT2ST significantly improves training efficiency-achieving up to 56% FLOPs compression-while maintaining or surpassing task performance.These results suggest MT2ST as a general-purpose solution for scalable and adaptive multi-task training.Although this work is general-purpose, it is especially suitable for multimodal settings such as VQA or vision-language retrieval, where auxiliary pretraining (e.g., masked language modeling or contrastive learning) often diverges from final objectives.We include a VQA case study and outline its efficiency for multimodal retrieval in 4.