Context-Aware Retrieval Augmented Generation Using Similarity Validation to Handle Context Inconsistencies in Large Language Models

Large Language Models (LLMs) have transformed natural language processing by offering human-like responses. However, issues such as incorrect information (hallucinations) and errors in specific subject areas remain, especially in Retrieval Augmented Generation (RAG) systems. This study introduces a Context-Aware Retrieval Augmented Generation (CA-RAG), which simplifies the process by removing the need to separately find relevant chunks of a document. Instead, after dividing the document into chunks, both the question and chunks are directly given to the LLMs to produce answers. The method then focuses on improving the answers through additional post-processing, aiming to reduce errors and make the answers more relevant to the question. To evaluate the effectiveness of CA-RAG, two scenarios have been designed. Scenario 1 involved experiments using widely adopted and recognized benchmark datasets, such as TriviaQA, Natural Questions, AmbigQA and Stanford Question Answering Dataset (SQuAD). In this context, the proposed CA-RAG method, combined with similarity measure (either cosine similarity or dot product) between generated answers and chunks, achieved the highest F1-score in TriviaQA and AmbigQA. Scenario 2 tested CA-RAG robustness using a custom dataset comprising domain-specific and unstructured documents. Results from automated and manual evaluations revealed that CA-RAG with post-processing consistently outperformed traditional RAG. These findings highlight the critical role of post-processing techniques and similarity measures in improving the accuracy and relevance of generated answers. CA-RAG demonstrates strong potential as a reliable and versatile solution for Retrieval Augmented Generation tasks across diverse datasets and domains.