AWeCita: Generating Answer with Appropriate and Well-grained Citations Using LLMs

Large language models (LLMs) excel in various Natural Language Processing tasks but struggle with hallucinations, leading to potentially misleading responses. Researchers have extensively explored LLMs’ citation practices. However, existing efforts often overlook the crucial aspects of the appropriateness and granularity of citation, which are vital for mitigating hallucination and enhancing interpretability. To bridge this gap and improve the quality of citations, we propose the Generating Answers with Appropriate and Well-grained Citations using LLMs task (AWeCita), with a focus on citing appropriately with a well granularity. Based on the traditional evaluation metrics of answer accuracy and citation correctness, we introduce two new evaluation metrics, citation appropriateness and citation granularity, to assess LLMs’ performance on this task more comprehensively and accurately. We conduct a series of exploratory experiments on ASQA and ELI5 datasets. The experimental results show that, AWeCita outperforms traditional tasks in the metric of citation granularity, most of our methods show a certain advantage incitation appropriateness, however, the improvement towards well-grained citation affects the quote-level citation correctness.