T5Score: A Methodology for Automatically Assessing the Quality of LLM Generated Multi-Document Topic Sets

Using LLMs for Multi-Document Topic Extraction has recently gained popularity due to their apparent high-quality outputs, expressiveness, and ease of use.However, most existing evaluation practices are not designed for LLM-generated topics and result in low interannotator agreement scores, hindering the reliable use of LLMs for the task.To address this, we introduce T 5 Score, an evaluation methodology that decomposes the quality of a topic set into quantifiable aspects, measurable through easy-to-perform annotation tasks.This framing enables a convenient, manual or automatic, evaluation procedure resulting in a strong interannotator agreement score.To substantiate our methodology and claims, we perform extensive experimentation on multiple datasets and report the results. 1