Variation-aware proxy learning for semantic segmentation

In semantic segmentation, accurately modeling intra-class variation is essential for capturing fine-grained details and resolving ambiguity near class boundaries. While existing proxy-based embedding methods represent each class with a single prototype, they struggle to reflect diverse intra-class structures, especially in complex scenes. In this paper, we propose a novel representation learning framework called Variation-Aware Proxy Learning, which introduces a representative proxy to encode shared class semantics and multiple variation vectors to capture fine-grained intra-class variations. These components are integrated through a factorized similarity score, enabling more expressive and discriminative embedding structures. To further enhance learning in ambiguous regions, we introduce focal modulation and design a new Compositional Similarity Loss composed of attraction and repulsion terms that adaptively amplify the contribution of hard examples. Our method is model-agnostic and requires no additional inference-time cost. Extensive experiments across multiple segmentation benchmarks—Cityscapes, COCO-Stuff10k, iSAID and ADE20K—and diverse backbones including CNNs and Transformers, demonstrate consistent improvements in mIoU and boundary precision, particularly in challenging regions with high intra-class variability. • Compositional Similarity Loss enables the learning of representative proxies and variation vectors. • Factorized similarity score achieves enhanced class separation with proxy integration. • Focal modulation adaptively emphasizes ambiguous regions with dynamic weighting. • Consistent segmentation gains across diverse real-time and non-real-time models.