Multimodal retrieval-augmented generation framework for visually rich knowledge in the architecture domain

Abstract Architectural design relies heavily on rich and multimodal knowledge—including text descriptions, detailed tables, and complex visual information—to inform creative and technical decision-making. However, effectively retrieving and generating meaningful insights from such diverse data sources remains challenging. In this study, we propose Multimodal Retrieval-Augmented Generation (M-RAG), a multimodal retrieval-augmented generation framework that integrates a Multimodal Large Language Model (MLLM) with a unified knowledge base via a shared semantic embedding space, dual-mode table decomposition (text and image), and adaptive query handling to enable traceable cross-modal retrieval and augmented answer generation. We use a multimodal embedding model to project text, table-rendered images, and diagrams into a common space. Quantitative evaluation on the dataset employed in the study reports average text–image similarity of 0.55 ± 0.07; we use an operational threshold θ = 0.5 to flag likely out-of-domain queries and trigger clarification, reducing low-relevance returns. We evaluate the M-RAG framework across multiple architectural scenarios—including reducing technical hallucinations in structural-engineering queries by returning supporting diagrams, enabling special-technical comparison reasoning, handling floorplans and sections, supporting regulation and compliance lookups, retrieving sustainable concept design images and explanations, facilitating material queries, and demonstrating depth-first and breadth-first exploration examples—showing its practical usefulness. By quantifying embedding alignment, retrieval performance, and an operational relevance threshold, M-RAG delivers a more reliable and traceable multimodal retrieval and generation capability for building engineering tasks.