Cross-modal domain generalization semantic segmentation based on fusion features

The primary techniques for domain generalization in semantic segmentation revolve around domain randomization and feature whitening. Although less commonly employed, methods based on cross-modality have demonstrated effective outcomes. This paper introduces enhancements to cross-modal feature alignment by redesigning the feature alignment module. This redesign facilitates alignment across different modalities by leveraging fusion features derived from both visual and textual inputs. These fusion features provide a more effective anchor point for alignment, enhancing the transfer of semantic information from textual to visual domains. Furthermore, the decoder plays a crucial role in the model as its ability to categorize features directly impacts the segmentation performance of the entire model. To enhance the decoder’s capability, this study employs the fusion features as the input for the decoder, with image labels providing the supervision. Experimental results indicate that our approach significantly enhances the model’s generalization capabilities. • We are the first to use fusion features as anchors for cross-modal alignment, thereby enhancing the model’s generalization capabilities. • We use the fusion feature as the input to the decoder and supervise it with labels to ensure it retains semantic information beneficial to vision. • Our method exhibits significant enhancements in generalization ability compared to previous cross-modal approaches.