Measuring the Fairness Gap Between Retrieval and Generation in RAG Systems using a Cognitive Complexity Framework

In this paper, we investigate the problem of quantifying fairness in Retrieval-Augmented Generation (RAG) systems, particularly for complex cognitive tasks that go beyond factual question-answering. While RAG systems have demonstrated effectiveness in information extraction tasks, their fairness implications for cognitively complex tasks - including ideation, content creation, and analytical reasoning - remain under-explored. We propose a novel evaluation framework that extends IR fairness metrics by incorporating centrality-based measures to account for influence of retrieved documents on generated output beyond ranking. Our framework evaluates RAG systems across various cognitive dimensions using two ranking approaches: lexical (BM25) and dense (BGE), and language models of varying sizes. Our findings provide insights into: (1) the propagation of fairness disparities from retrieval to generation phases, and (2) the variation in system performance across different cognitive dimensions.