Toward Unified Expertise: Learning a Single Vision Model From Diverse Perception

Multi-task learning (MTL) presents greater optimization challenges than single-task learning (STL) due to conflicting gradients across tasks. While parameter sharing promotes cooperation among related tasks, many tasks require specialized representations. To balance cooperation and specialization, we propose Mod-Squad (Chen et al. 2023), a modular transformer-based model composed of a "squad" of experts. Each task activates a sparse subset of experts through a differentiable matching process, guided by a novel mutual information-based loss. This modular structure avoids full backbone sharing and scales effectively with the number of tasks and dataset size. In this extended version, we generalize Mod-Squad to support multi-dataset pre-training, enabling joint learning across disjoint, single-task datasets (e.g., ImageNet, COCO, ADE20 K). This is achieved via a new formulation of the mutual information loss that unifies learning across heterogeneous sources. More importantly, while most prior work in large models has focused on efficiency, few have explored adjustable efficiency. In this study, we further evaluate the model's generalization to downstream tasks and introduce a set of efficient adaptation techniques that leverage Mod-Squad's modularity for flexible fine-tuning-enabling dynamic adjustment of model size, parameter count, and computational cost. Additionally, we present a hybrid adaptation scheme that combines these techniques to achieve favorable performance-efficiency trade-offs. In summary, Mod-Squad provides a robust foundation for sparse modular models that can learn from diverse supervision and datasets. Its emergent modularity enables strong generalization, decomposition into high-performing components, and rapid, resource-efficient adaptation for downstream applications.