Cross-Task Affinity Learning for Multitask Dense Scene Predictions

Multitask learning (MTL) has become prominent for its ability to predict multiple tasks jointly, achieving better per-task performance with fewer parameters than single-task learning. Recently, decoder-focused architectures have sig-nificantly improved multitask performance by refining task predictions using features from related tasks. However, most refinement methods struggle to efficiently capture both local and long-range dependencies between task-specific representations and cross-task patterns. In this paper, we introduce the Cross-Task Affinity Learning (CTAL) module, a lightweight framework that enhances task refinement in multitask networks. CTAL effectively captures local and long-range cross-task interactions by optimizing task affin-ity matrices for parameter-efficient grouped convolutions without concern for information loss. Our results demon-strate state-of-the-art MTL performance for both CNN and transformer backbones, using significantly fewer parame-ters than single-task learning.