LevelEval: Adaptive Pipeline for Evaluating LLM as a Judge - Analysis on Open LLMs as Judges

This paper presents LevelEval, an automated pipeline to evaluate LLMs as judges, mainly using state-of-the-art pretrained models. The research investigates on how open LLMs with fewer parameters compare to GPT-4. The LevelEval pipeline involves: i) generating outputs of different quality using a task-oriented LLM, ii) evaluating and scoring these outputs on a scale by an LLM judge using a rubric, iii) categorizing the scores into ranks to assess the LLM judges' performance. LevelEval is applied to four tasks: summarization, question answering, retrieval augmentation, and bias classification. The datasets are tailored for evaluation and can serve as benchmarks. Overall, the experiments highlight the effectiveness of LevelEval in evaluating LLMs and offer insights into the judging abilities of open LLMs with smaller parameters.