One Prompt to Rule Them All: Automated Curve and Image Retrieval from PDFs and Websites with Agentic Multimodal RAG

In the water industry, subject matter experts (SMEs) often manually review numerous documents and websites to find image-based information, such as chemical reaction curves and software manual screenshots for model setups. This process is time-consuming, inconsistent, and inefficient. This paper introduces a multimodal agentic Retrieval-Augmented Generation (RAG) tool to automate this task. RAG combines information retrieval systems with generative models to process and integrate multiple data types, including text, images, and tables. To achieve our goal, we designed a unique nested agentic multimodal RAG with three main large language model (LLM) agents. The first agent, although optional, conducts research in the absence of a library of PDFs and websites. For instance, it can locate top articles for a specific topic in the water industry such as water treatment removal curve. The second agent, a multimodal RAG specialist, extracts images, curves, tables, and text. Finally, the third agent digitizes the extracted curves into CSV format. This framework has successfully produced accurate and reliable results for multiple test cases. In one example, the framework automatically analyzed PDFs and websites and provided turbidity removal curves for alum from various resources. Another experiment focused on extracting both images and text from a software manual to answer questions about cost factors. This advanced automation framework leads to better decision-making, optimized performance, and increased efficiency in managing water infrastructure. Moreover, it significantly reduces the manual effort required by subject matter experts (SMEs), enabling them to focus on higher-level analytical tasks.