Multi-Modal Prompts With Primitives Enhancement for Compositional Zero-Shot Learning

Compositional zero-shot learning (CZSL) aims to recognize novel compositions of known attributes and objects without requiring additional training data. Recent CZSL methods based on vision-language models(e.g., CLIP) suffer from relying solely on text prompts and neglecting the crucial primitive features within compositions, which limits generalization to unseen compositions. To overcome these limitations, we propose a Multi-modal Prompt and Primitives Enhancement method, termed MPPE, which incorporates two key aspects. First, MPPE introduces both text and visual prompts. The text prompts consist of the composition and its corresponding attribute and object prompts, while the visual prompts leverage image masks generated by the segment anything model (SAM). These masks are integrated via an additional Alpha branch to strengthen the CLIP visual encoder to focus on regions of interest within the image. Second, we design a primitives enhancement (PE) module based on cross-attention, which refines attribute and object features obtained from the CLIP text encoder, thereby enriching the representation of novel composition features. Extensive experiments demonstrate the effectiveness of our approach, achieving state-of-the-art performance on three widely-used CZSL benchmarks in both closed-world and open-world CZSL scenarios. Codes are available at https://github.com/YtJin-git/MPPE.