MR-Former: Improving universal image segmentation via refined masked-attention transformer

Mask-based universal image segmentation architectures, which focus attention within the mask region, have significantly enhanced both segmentation accuracy and speed, becoming a key component of universal image segmentation frameworks. However, existing methods often generate masks from the predicted probability map using a hard threshold, which may introduce biased guidance due to missing or inaccurate object predictions. To address this issue, we propose a novel mask refinement method, called MR-Former. The method divides the segmentation output into two categories: ‘object’ and ‘non-object’ masks. For ‘object’ masks, we further refine the mask into interior and edge regions, and apply independent attention branches to learn their respective attention distributions, which are then adaptively fused. For ‘non-object’ masks, we introduce a mask focusing technique that objects key regions to ensure attention is concentrated on high-probability areas, effectively eliminating irrelevant distractions. Experimental results demonstrate that MR-Former, with a ResNet-50 backbone, significantly improves segmentation performance across multiple state-of-the-art architectures, including Mask2Former, OneFormer, and PEM. On the Cityscapes dataset, MR-Former achieves improvements of up to 0.9 PQ, 0.7 AP, and 1.3 mIoU with a maximum Params increase of only 1.4%; on the ADE20K dataset, it shows improvements of up to 1.1 PQ, 1.4 AP, and 1.3 mIoU accompanied by a maximum FLOPs increment of just 1G; and on the Mapillary Vistas dataset, it demonstrates improvements of up to 1.1 PQ and 1.2 mIoU with a maximum FPS loss of merely 3.4.