CrossTaskFormer: Bidirectional cross-task attention for joint brain tumor segmentation and classification

Accurate brain tumor MRI decision support requires both reliable lesion localization and consistent tumor-type prediction, yet many deep learning pipelines optimize segmentation and classification separately or with limited task interaction. We propose CrossTaskFormer, a unified multi-task framework that coordinates these objectives via learnable bidirectional cross-task attention, enabling structured exchange of localization cues and subtype-discriminative semantics across decoder stages. The model uses transfer learning for multi-scale feature extraction, incorporates an attention-based bottleneck to capture global context, and applies progressive cross-task fusion to improve boundary stability and diagnostic consistency. We evaluate CrossTaskFormer under a deployment-oriented inference protocol that includes ensembling, test-time augmentation, and morphological mask refinement. On the BRISC contrast-enhanced MRI dataset spanning four diagnostic categories, CrossTaskFormer achieves 99.30% classification accuracy while maintaining strong segmentation performance (Dice = 0.8925, IoU = 0.8122) under the full inference pipeline, while the raw base model already attains 99.23% classification accuracy with Dice = 0.8866 and IoU = 0.8032. Error analysis indicates that residual misclassifications are infrequent and largely confined to clinically plausible inter-class confusions, whereas segmentation degradations occur primarily in small-lesion cases where overlap metrics are inherently sensitive to minor boundary deviations. These results suggest that explicit cross-task coupling can simplify integrated pipelines while producing coherent outputs promising for integrated neuro-oncology decision support under controlled benchmarking conditions.