MELLM - An Automatic Evaluation Framework for LLM Without Human Supervision

Recently, the landscape of large language models (LLMs) has been rapidly evolving, accompanied by a surge in new benchmarks and datasets designed o evaluate their performance. Despite many models claim to have surpassed ChatGPT on specific metrics and benchmarks, widespread skepticism persists. This skepticism arises from the recognition that evaluating LLMs is a customized matter, requiring a simplified yet reliable and unbiased evaluation process that can be easily accessed by everyone. To address this need, we introduce a novel benchmarking framework called Mutual-Evaluation-of-LLMs (MELLM). This framework revolutionizes the evaluation process by allowing multiple LLMs to serve both as examiners and examinees, synthesizing their results into a comprehensive total score. The key advantage of MELLM lies in its effortless extensibility, enabling the seamless integration of various LLMs as both examiners and examinees. Furthermore, we devise strategies to generate questions from seeds, ensuring a lightweight and equitable evaluation process. The results indicate that scores calculated by MELLM demonstrate strong linear correlation with both GPT-4 and human scoring. Codes and parts of questions and results are available at: https://github.com/dongrixinyu/JioNLP.