A unified model for image understanding and generation based on cross-modal representation learning

Unified image understanding and text-conditioned image generation require models to coordinate discriminative semantic representations with generation-compatible latent representations. Existing approaches often rely on either strongly shared representations, which may cause task interference, or separated branches, which may weaken semantic transfer. This paper proposes CSB-UG, a unified understanding–generation framework based on a cross-modal semantic bridge. CSB-UG projects visual features, textual semantics, and generative latents into an intermediate bridge space, while task-adaptive adapters preserve task-specific transformations. It further introduces controlled bidirectional collaboration and a patch-token consistency objective to enhance semantic guidance, latent feedback, and local alignment. Experiments on MS-COCO Captions, VQAv2, Visual Genome, and GenEval evaluate retrieval, visual question answering, captioning, text-to-image generation, generation alignment, and fine-grained semantic consistency. Results show that CSB-UG achieves competitive performance across the evaluated understanding and generation tasks, with improvements in retrieval, captioning, generation alignment, and local semantic consistency. Qualitative analyses further suggest improved constraint preservation and object-, attribute-, and relation-level alignment, while failure cases show that complex spatial relations, counting, long-prompt following, and rare attributes remain challenging. These results indicate that bridge-based semantic coordination is a useful design choice for unified image understanding and generation under the evaluated settings.

Authors

Institutions

Publication Details

Journal
Discover Artificial Intelligence
Published
2026-11-17
DOI
https://doi.org/10.1007/s44163-026-02141-y
Primary Topic
Multimodal Machine Learning Applications
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

A unified model for image understanding and generation based on cross-modal representation learning

Xiaoping Gu
1 links
Discover Artificial Intelligence
Multimodal Machine Learning Applications
article

A unified model for image understanding and generation based on cross-modal representation learning

Xiaoping Gu
article en

Abstract

Unified image understanding and text-conditioned image generation require models to coordinate discriminative semantic representations with generation-compatible latent representations. Existing approaches often rely on either strongly shared representations, which may cause task interference, or separated branches, which may weaken semantic transfer. This paper proposes CSB-UG, a unified understanding–generation framework based on a cross-modal semantic bridge. CSB-UG projects visual features, textual semantics, and generative latents into an intermediate bridge space, while task-adaptive adapters preserve task-specific transformations. It further introduces controlled bidirectional collaboration and a patch-token consistency objective to enhance semantic guidance, latent feedback, and local alignment. Experiments on MS-COCO Captions, VQAv2, Visual Genome, and GenEval evaluate retrieval, visual question answering, captioning, text-to-image generation, generation alignment, and fine-grained semantic consistency. Results show that CSB-UG achieves competitive performance across the evaluated understanding and generation tasks, with improvements in retrieval, captioning, generation alignment, and local semantic consistency. Qualitative analyses further suggest improved constraint preservation and object-, attribute-, and relation-level alignment, while failure cases show that complex spatial relations, counting, long-prompt following, and rare attributes remain challenging. These results indicate that bridge-based semantic coordination is a useful design choice for unified image understanding and generation under the evaluated settings.

Discover Artificial IntelligenceVol. 6(1)
Zhixing College of Hubei University (CN)
Multimodal Machine Learning Applications
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon