MO-SHW: hierarchy-aware multi-objective optimization for open-world segmentation

The exploitation of hierarchical information by vision models has shown significant benefits in various segmentation tasks. However, this remains largely unexplored in open-world scenarios, where models must cope with unknown, evolving, and underrepresented labeled class spaces. Most existing hierarchy-aware segmentation approaches are not readily applicable to open-world settings. This is primarily because they rely on architectural modifications that are incompatible with the design constraints of open-world models. Moreover, hierarchy-aware losses are challenging to integrate into such pipelines, as they often conflict with task-specific objectives and exacerbate optimization complexity in already multi-objective training environments. In this work, we demonstrate that hierarchy-aware losses can be effectively leveraged in open-world models when optimized under a multi-objective learning framework. Specifically, we show that gradient-based multi-objective optimization methods, such as multi-objective gradient descent (MOGD), are well-suited for jointly optimizing hierarchical and task-specific objectives, leading to better overall performance. To support this, we propose SHW, a novel hierarchy-aware loss function based on the Wasserstein distance. SHW is lightweight, model-agnostic, and encourages intra-class compactness and inter-class separation across multiple semantic levels. The integration of SHW with MOGD yields a general, model-agnostic framework that enables the effective exploitation of semantic hierarchies in open-world segmentation tasks, improving the performance of several recent methods.