Learning to Fuse Retrieval Signals: A Lightweight Meta Relevance Classifier for RAG Pipelines

Retrieval-Augmented Generation (RAG) systems typically rely on two-stage pipelines that first retrieve candidate documents using fast bi-encoders and then rerank them using more precise but slower cross-encoders. But this technique usually delivers irrelevant results to downstream modules, which deteriorate performance. To address this, we introduce a lightweight, model-agnostic Meta Relevance Classifier that fuses signals from multiple retrieval components to predict passage relevance, enabling both precision-oriented filtering and improved reranking. It operates on precomputed scores without modifying underlying model architecture. Experiments across four BEIR subsets demonstrate consistent improvements in filtering (F1) and reranking (nDCG@10) performance. Even our basic 2 feature classifier provides substantial gains across BEIR sets: enhancing filtering by 8.6% on HotpotQA and 19.0% on FiQA over cross-encoder thresholding and boosting reranking by 5.0% on NQ and 11.6% on FiQA over traditional two stage retrieval. This classifier is extremely data efficient, taking only 100-500 training queries to achieve strong performance and remains effective when adding other signals such as BM25. This is a practical solution to a primary RAG pipeline bottleneck, enhancing retrieval effectiveness with low computational and implementation burdens.