GEM-RAG: Graphical Eigen Memories for Retrieval Augmented Generation

The ability to form, retrieve, and reason about memories in response to stimuli is central to general intelligence, enabling learning, adaptation, and insight. Large Language Models (LLMs), when given proper memories or context, can reason and respond effectively. However, they still struggle to optimally encode, store, and retrieve memories, a limitation that constrains their full potential as specialized AI agents. Retrieval Augmented Generation (RAG) seeks to address this by enriching LLMs with in-context examples. Inspired by human memory, we introduce Graphical Eigen Memories for Retrieval Augmented Generation (GEM-RAG), which tags information with LLMgenerated “utility” questions, links information together in a graph by similarity, and uses eigendecomposition to form higherlevel summary information. This approach not only enhances RAG tasks but also offers a way to explore text data sets. Using UnifiedQA, GPT-3.5 Turbo, SBERT, and OpenAI text encoders, we show that GEM-RAG outperforms state-of-the-art RAG methods on two standard QA tasks and discuss its implications for robust RAG systems.