Decomposed Modular RAG: Task-Aware Multi-Step Retrieval and Reasoning for Small-Scale Language Models

Retrieval-Augmented Generation (RAG) has emerged as a common solution to factual inaccuracies in large language models by incorporating external knowledge through retrieval. However, standard RAG methods are primarily designed for large-scale models and struggle when applied to small language models (SLMs), particularly in multi-hop question answering tasks that require complex reasoning. We propose Decomposed Modular RAG (DM-RAG), a framework tailored for SLMs that decomposes the RAG process into modular tasks, including decomposition, iterative retrieval, reasoning, and verification. A State Tracker compresses and manages reasoning context across steps, reducing memory load and enhancing consistency. To support training, we adopt a teacher-student distillation approach, where a large model provides question decomposition and chain-of-thought reasoning traces for supervising a unified small model. Our method enables end-to-end inference using a single model with task-aware prefixing. Experiments on HotpotQA, MuSiQue, and 2WikiMultihopQA demonstrate that DM-RAG significantly improves accuracy and reduces hallucination rates compared to existing RAG baselines, without significantly increasing computational cost. This framework offers a scalable path for deploying multi-step RAG systems on resource-constrained devices.