Evaluating LLMs’ Assessment of Mixed-Context Hallucination Through the Lens of Summarization

With the rapid development of large language models (LLMs), LLM-as-a-judge has emerged as a widely adopted approach for text quality evaluation, including hallucination evaluation.While previous studies have focused exclusively on single-context evaluation (e.g., discourse faithfulness or world factuality), realworld hallucinations typically involve mixed contexts, which remains inadequately evaluated.In this study, we use summarization as a representative task to comprehensively evaluate LLMs' capability in detecting mixedcontext hallucinations, specifically distinguishing between factual and non-factual hallucinations.Through extensive experiments across direct generation and retrieval-based models of varying scales, our main observations are:(1) LLMs' intrinsic knowledge introduces inherent biases in hallucination evaluation; (2) these biases particularly impact the detection of factual hallucinations, yielding a significant performance bottleneck; and (3) the fundamental challenge lies in effective knowledge utilization, balancing between LLMs' intrinsic knowledge and external context for accurate mixed-context hallucination evaluation. 1