CiteLab: Developing and Diagnosing LLM Citation Generation Workflows via the Human-LLM Interaction

The emerging paradigm of enabling Large Language Models (LLMs) to generate citations in Question-Answering (QA) tasks is lacking in a unified framework to standardize and fairly compare different citation generation methods, leading to difficulties in reproduction and innovation.Therefore, we introduce Citeflow, an open-source and modular framework fostering reproduction and the implementation of new designs.Citeflow is highly extensible, allowing users to utilize four main modules and 14 components to construct a pipeline, evaluate an existing method, and understand the attributing LLM-generated contents.The framework is also paired with a visual interface, Citefix, facilitating case study and modification of existing citation generation methods.Users can use this interface to conduct LLM-powered case studies according to different scenarios.Citeflow and Citefix are highly integrated into the toolkit CiteLab, and we use an authentic process of multiple rounds of improvement through the Human-LLM interaction interface to demonstrate the efficiency of our toolkit on implementing and modifying citation generation pipelines.