MinosEval: Distinguishing Factoid and Non-Factoid for Tailored Open-Ended QA Evaluation with LLMs

Open-ended question answering (QA) is a key task for evaluating the capabilities of large language models (LLMs).Compared to closedended QA, it demands longer answer statements, more nuanced reasoning processes, and diverse expressions, making refined and interpretable automatic evaluation both crucial and challenging.Traditional metrics like ROUGE and BERTScore struggle to capture semantic similarities due to different patterns between model responses and reference answers.Current LLM-based evaluation approaches, such as pairwise or listwise comparisons of candidate answers, lack intuitive interpretability.While pointwise scoring of each response provides some descriptions, it fails to adapt across different question contents.Most notably, existing methods overlook the distinction between factoid and non-factoid questions.To address these challenges, we propose MinosEval, a novel evaluation method that first distinguishes open-ended questions and then ranks candidate answers using different evaluation strategies.For factoid questions, it applies an adaptive key-point scoring strategy, while for nonfactoid questions, it uses an instance-aware listwise ranking strategy.Experiments on multiple open-ended QA datasets, including self-built ones with more candidate responses to complement community resources, show that MinosEval better aligns with human annotations and offers more interpretable results. FactoidQuestion: Did the Magellanic Flotilla use a sextant to measure latitude and longitude while traveling around the world?Answer: Magellan's voyage around the world took place from 1519 to 1522, and the principle of the sextant was proposed by Isaac Newton, who was born on January 4, 1643 and the use of a sextant was not possible.Question: List the five tallest mountains on Earth and the average of their peaks.Answer: the five tallest mountains on earth and their altitudes are: 1.Mount Everest: 8848 meters average of their altitudes is (8848+8611+8586+8516+8485)/5 = 8609.2meters.