Nugget-based Annotation Protocol and Tool For Evaluating Long-form Retrieval-Augmented Generation

Retrieval-augmented generation (RAG) summarizes retrieved documents into a text passage that fulfills the information need expressed by the user. Such generated responses should faithfully distill the relevant information and provide sufficient attribution back to the source documents. Nugget-based evaluation was proposed for text summarization and has been adapted to evaluate RAG output in recent shared tasks such as 2024 TREC RAG, BioGen, and NeuCLIR tracks. However, annotating such detailed and nuanced information is complex and errorful. Multiple pieces of information need to be labeled, extracted, linked, and cross-referenced. In this work, we present an annotation protocol and tool tailored to collecting information for evaluating RAG systems. Our tool has four steps: nugget creation, nugget revision, document support assessment, and finally, nugget alignment. Each step aims to minimize the annotator's cognitive load, improving the efficiency and reliability.