A survey of multimodal composite editing and retrieval

Abstract In the real world, where information is abundant and diverse across different modalities, understanding and utilizing various data types to improve retrieval systems is a key focus of research. Multimodal composite retrieval integrates diverse modalities such as text, image and audio to provide more accurate, personalized, and contextually relevant results. However, alongside retrieval, multimodal composite editing plays a crucial role in enabling users to refine or modify retrieved content through intuitive interactions, which enhances the overall effectiveness of multimodal systems. The task of multimodal composite editing is becoming increasingly critical due to its applications in various domains, including creative industries, education, and user-driven content modification. A comprehensive evaluation and usage guide is needed to fully assess its capabilities and limitations, since it complements and extends the functionalities provided by multimodal retrieval systems. To facilitate a deeper understanding of this promising direction, this survey explores multimodal composite editing and retrieval in depth, covering image-text composite editing, image-text composite retrieval, and other multimodal composite retrieval. In this survey, we systematically organize the application scenarios, methods, benchmarks, experiments, and future directions. Multimodal learning has gained significant popularity in the era of large AI models, as demonstrated by the growing number of surveys in multimodal learning and vision-language models with Transformers. To the best of our knowledge, this survey is the first comprehensive review of the literature on multimodal composite retrieval, which is a timely complement of multimodal fusion to existing reviews. Moreover, this paper bridges the gap between large model architectures and their applications in both retrieval and editing tasks, highlighting their intertwined roles in advancing multimodal systems.