Cross-modal content generation and semantic alignment of digital media based on CLIP-LLaVA

This paper proposes a cross-modal content bidirectional generation framework based on the CLIP-LLaVA model and SDXL to resolve semantic mismatches between text and images and insufficient fine-grained semantic alignment in cross-modal digital media generation. The method involves extracting global visual-language embeddings via CLIP, combining DETR detection with region-level feature extraction, and injecting visual prefixes into LLaVA to achieve instruction-driven structured text semantic generation. The structured text semantics are then mapped to SDXL conditional vectors, and images are generated under the guidance of UNet cross-modal conditional attention. By combining region importance weighting and prompt correction mechanisms, macro-semantic consistency and local attribute alignment are ensured. The experiment was validated on a self-built news social media subset and Visual Genome, with a CLIP-score of 0.33 ± 0.02 and a region-level alignment accuracy of 0.81. The experimental results demonstrate the effectiveness of the proposed framework in maintaining semantic consistency, region-level correspondence, and generation fidelity across multimodal generation tasks, indicating its potential as a feasible solution for high-precision digital media content generation.