Hierarchical Context Learning of object components for unsupervised semantic segmentation

Unsupervised Semantic Segmentation (USS) aims to learn semantically rich and dense representations without relying on labels. Recent advances in self-supervised learning have demonstrated the potential of pretrained vision transformers to capture patch-level semantic information, offering a promising direction to USS. However, existing methods face challenges in constructing a discriminative spatial token embedding space that consistently and effectively represents the well-structured semantic relationships among object components. Inspired by Edwin Hancock’s pioneer work on hierarchical pattern analysis, we highlight the critical role of hierarchical context to overcome this limitation. By modeling spatial relationships at multiple levels of granularity, hierarchical context helps align related object parts while distinguishing them across semantic groups. Based on this insight, we introduce Hierarchical Context Learning (HCL), a novel approach for USS that enhances semantic consistency by integrating hierarchical context. HCL incorporates a novel parallel multi-level vision transformer backbone to aggregate multi-level contextual information into object component tokens. To uncover the semantic structure of objects, we propose Momentum-based Global Foreground–Background Clustering (MoGoClustering) to cluster object components into coherent semantic groups and then calculate their semantic centroids. To enforce intra-group semantic consistency and maximize inter-group separation across spatial scales, we design a foreground–background-aware contrastive loss based on MoGoClustering. Our method achieves state-of-the-art performance on the COCO-Stuff and Pascal VOC datasets, demonstrating its ability to learn robust, context-aware, and discriminative object component semantics for USS. The code is available at: https://github.com/dbaofd/HCL .