Enriching Retrieval-AugmentedGeneration with Non-TextualInformation to Support ScientificWriting

Retrieval-Augmented Generation (RAG) systems have primarily focused on leveraging textualinformation, often overlooking the insight offered by non-textual elements such as figures, tables,equations, and code snippets. Furthermore, research on this topic remains sparse, withfew comprehensive approaches to tackling the integration of such elements into RAG pipelines.This thesis addresses this gap by proposing a method to enrich RAG systems with non-textualinformation, with a particular focus on research papers. The objective of this work is to enhance the capabilities of an academic research assistant RAGsystem by integrating non-textual content extracted from scientific documents. The proposedmethod involves a multi-step approach: extraction of non-textual components from diverse researchpapers using tools such as GROBID and Marker, processing and contextualization ofsaid components through classification and the extraction of related information from the papersand ultimately converting them into textual information using multimodal Large Language Models(LLMs) and iterative prompt engineering. This methodology is tested across multiple modelsprovided by OpenAI and Gemini to demonstrate its adaptability. The core result of this thesis is a complete, modular pipeline that transforms non-textual elementsinto meaningful textual descriptions, enabling their use within existing RAG systems, with a focuson academic writing. The framework supports customization, with each subsystem (extraction,processing, contextualization, and summarization) designed to be independently upgradable.Additionally, the thesis provides validated starting points for each subsystem: a reliable methodfor extracting non-textual elements, an effective approach to processing them accordingly throughsegmentation and classification, contextualizing them using related textual information, and a setof tested prompts to guide LLMs during summarization. In conclusion, this work attempts to lay the groundwork for future research into more efficientintegration of non-textual information into RAG systems of any type. It does so by offering asimple but scalable pipeline and proposing starting points for each step.