A unified model for image understanding and generation based on cross-modal representation learning
Unified image understanding and text-conditioned image generation require models to coordinate discriminative semantic representations with generation-compatible latent representations. Existing approaches often rely on either strongly shared representations, which may cause task interference, or separated branches, which may weaken semantic transfer. This paper proposes CSB-UG, a unified understanding–generation framework based on a cross-modal semantic bridge. CSB-UG projects visual features, textual semantics, and generative latents into an intermediate bridge space, while task-adaptive adapters preserve task-specific transformations. It further introduces controlled bidirectional collaboration and a patch-token consistency objective to enhance semantic guidance, latent feedback, and local alignment. Experiments on MS-COCO Captions, VQAv2, Visual Genome, and GenEval evaluate retrieval, visual question answering, captioning, text-to-image generation, generation alignment, and fine-grained semantic consistency. Results show that CSB-UG achieves competitive performance across the evaluated understanding and generation tasks, with improvements in retrieval, captioning, generation alignment, and local semantic consistency. Qualitative analyses further suggest improved constraint preservation and object-, attribute-, and relation-level alignment, while failure cases show that complex spatial relations, counting, long-prompt following, and rare attributes remain challenging. These results indicate that bridge-based semantic coordination is a useful design choice for unified image understanding and generation under the evaluated settings.
Authors
- Xiaoping Gu
Institutions
- Zhixing College of Hubei University (CN)
Publication Details
- Journal
- Discover Artificial Intelligence
- Published
- 2026-11-17
- DOI
- https://doi.org/10.1007/s44163-026-02141-y
- Primary Topic
- Multimodal Machine Learning Applications
- Type
- article
- Field-Weighted Citation Impact
- 0.00