Evaluation Framework for Large Language Models: Assessing Factual Consistency and Answer Relevancy in Question-Answering Tasks

The research evaluates a natural language understanding model for question-answering tasks using factual consistency and answer relevancy metrics. A pre-trained Natural Language Inference (NLI) model is used for factual consistency, while sentence-transformer-based cosine similarity measures answer relevancy. The dataset consists of context-question-answer pairs, and results indicate strong performance in factual consistency but challenges in negation and contradictions. The model also demonstrates robust answer relevancy, with some variability in indirect responses. This evaluation framework guides future improvements for more accurate question-answering systems.