Multi-Document Question Answering with Lightweight Embeddings-based Document Reranker

In the era dominated by the pre-trained Large Language Models (LLMs), Multi-document Question Answering (MDQA) systems play a vital role in efficiently accessing relevant knowledge from external databases to render accurate and useful answers, as the modern LLMs tend to hallucinate answers and suffer from static knowledge base. Since the quality of the MDQA systems' output depends on the quality of the evidence documents used to form the answer, the retriever component plays a crucial role in the performance of the whole system. However, the desire to improve the quality of the retrievers and, therefore, the overall performance of the MDQA systems has led to the emergence of the dense retrievers that use LLMs with huge model sizes, which might be impractical to use in low-resource conditions despite their superior performance. To address this challenge, this paper proposes an MDQA system featuring an efficient document retrieval approach and the latest GPT-4o. Our system, built upon the Visconde architecture, comprises three interconnected components: Question Decomposition, Document Retrieval, and Aggregation. The Document Retrieval Component, crucial for system performance, utilizes a two-step filtering process, initially using the BM25 search engine and subsequently reranking documents with MixedBread's state-of-the-art lightweight embedding model. We evaluate our system on the StrategyQA dataset, achieving superior results in terms of the accuracy of the answers and the quality of the retrieved evidence documents compared to Visconde, which uses a retrieval model with ten times more parameters. The proposed solution balances computational efficiency with performance, making a step towards exploring efficient MDQA systems.