Retrieval-Augmented Generation with Charts : Improving retrieval for images of charts

Recent advancements in Retrieval-Augmented Generation (RAG) systems have enabled large language models (LLMs) to better access and utilize external knowledge, yet the integration of non-textual data, particularly charts, remains underexplored. This thesis investigates strategies for preprocessing and embedding chart images in multimodal RAG systems to improve retrieval and generation performance. Three embedding methods, text summary, dual storage, and page screenshot, were developed and evaluated on two datasets: the chart-only CHARTQA and the newly created real-world financial report based COINQA dataset. Results show that while all strategies perform similarly on chart-only data, the page screenshot embedding significantly outperforms other methods in retrieval tasks for mixed text-and-chart documents, achieving more than twice the retrieval performance on COINQA. These findings highlight the effectiveness of page screenshot embeddings for multimodal retrieval, particularly in real-world financial contexts. Additionally, we reveal a need for further research on evaluation metrics in multimodal RAG systems.