Knowledge Disitillation in Mask2Former for Semantic Segmentation

In this paper, we propose a knowledge distillation framework for Mask2Former to enhance the performance of lightweight segmentation models. The proposed method integrates feature-level and attention-level distillation strategies by transferring knowledge from a larger teacher model (ResNet-101) to a compact student model (ResNet-18). To facilitate effective feature map transfer, a Channel Expansion Module (CEM) is introduced for channel alignment between the teacher and student backbones. Furthermore, a novel Self-Attention Map (SAM) distillation method is employed to align attention maps in the transformer decoder via a query-matching mechanism based on cosine similarity and the Hungarian algorithm. Experiments on the Cityscapes and ADE20K datasets demonstrate the effectiveness of the proposed approach. On Cityscapes, the student model with distillation achieves 79.24% mIoU (SS), a +2.14% improvement over the baseline. On ADE20K, the feature map distillation alone improves the student model by +1.70% mIoU. These results highlight that the proposed framework significantly enhances segmentation performance while maintaining model compactness, making it suitable for deployment in resource-constrained environments.