Multimodal Retrieval and Fusion Framework (MRaFF)

Retrieval-Augmented Generation (RAG) systems have traditionally focused on enhancing text generation by retrieving relevant textual information. While some recent RAG methods support multimodal retrieval, these approaches often involve pre-trained models or rely on pre-existing image-text associations to integrate visual content. In contrast, our proposed method introduces a multimodal RAG framework that integrates images without requiring pre-training with Large Language Models (LLMs). Our method begins by loading documents in formats such as PDF, PowerPoint, or Word and extracting both text and images separately. Then text is embedded using an LLM to generate vector representations, while images undergo quality checks to ensure validity. Each image is linked to text vectors through a fusion mechanism that matches their locations in the document. The combined metadata and vectors are then processed by the LLM, enabling the system to retrieve and generate responses that include both relevant text and images. This approach extends RAG systems beyond text-only retrieval, offering a powerful way to integrate visual content into generated responses. To the best of our knowledge, this is the first work to integrate images within a RAG framework with out the need of pre-training. Experimental results demonstrate that the framework achieves strong performance in generating accurate and context-rich multimodal responses.