Impact of Ensemble of Vector Embeddings on Speculative Retrieval Augmented Generation

Recent developments in Natural Language Processing (NLP) have impacted all fields. Retrieval Augmented Generation (RAG) has played significant part in resent advancements in NLP applications. RAG architecture combines the powers of retrieval-based methods and generative models to increase the relevance and accuracy of generated texts. This paper deals with the performance comparison on speculative RAG, recent development in RAG systems, with different state-of-the-art vector embedding models and their ensembles. Two approaches were adopted to carry out this research: one-to-one mapping context with response-rationale and mapping of clustered contexts with response and rationale. The performance measures semantic answer similarity (SAS), and Rouge score on descriptive-type NCERT books dataset. State-of-the-art vector embedding models, BAAI/bge-base-en, thenlper/gte-base and intfloat/e5-base-v2, were used individually and to create average and weighted average ensemble models.The responses of the systems corresponding to the 200 questions used for evaluation were validated against the NCERT Solution dataset. The result showed that the ensemble of vector embedding models outperforms state-of-the-art embedding models with weighted average ensemble achieving the highest accuracy. Further, the clustered context approach consistently outperforms the one-to-one mapping approach with noticeably higher SAS and Rouge metric values corresponding to each vector embedding method employed. For weighted average ensemble, the clustering approach improves the semantic answer similarity value by 8.24% over the one-to-one mapping approach. The performance of the systems may be improved by optimizing the weights of the weighted average ensemble and better clustering approaches. Various institutions using RAG systems for descriptive question-answer solutions may benefit from employing advancement in RAG systems.