LENS: Layers of Evaluation of Hallucination in GenAI Systems

Large Language Models (LLMs) and Vision-Language Models (VLMs) demonstrate remarkable capabilities but remain vulnerable to hallucinations—producing plausible yet factually incorrect content—with error rates ranging from as low as 1.47% in clinical applications [1] to as high as 75% in domain-specific queries [2]. Despite growing attention, existing hallucination evaluation frameworks remain insufficient to meet critical needs. Through a comprehensive survey of over 100 evaluation methods spanning six methodological paradigms (probe-based, adversarial testing, causal intervention, uncertainty-guided, internal state analysis, and online evaluation), we identify fundamental limitations: current approaches fall short of enabling objective model comparison, providing diagnostic insights into failure modes, supporting domain-evolving benchmark construction, and guiding targeted mitigation strategies. Our analysis reveals that evaluations remain fragmented, operating either horizontally—comparing models across tasks and domains—or vertically—probing reasoning chains within single outputs. This fragmentation limits holistic assessment: horizontal evaluations provide breadth but risk superficiality, while vertical assessments deliver depth but lack generalizability. Moreover, we identify five critical gaps: (1) dimensional poverty reducing hallucinations to binary metrics, (2) failure to integrate horizontal breadth with vertical depth, (3) metacognitive blind spots overlooking when models should seek external verification, (4) adaptability crisis from static benchmarks, and (5) transparency deficits providing scores without actionable insights. Beyond surveying the landscape, this paper articulates eight fundamental challenges confronting comprehensive evaluation—from epistemological difficulties in defining ground truth and computational complexity of scaling assessment, to attribution opacity obscuring causal mechanisms and dynamic knowledge evolution rendering benchmarks obsolete. These challenges span multimodal complexity, scale and diversity requirements, adversarial robustness, and human alignment considerations. We present LENS (Layers of Evaluation of Hallucination in GenAI Systems), a unified framework addressing these gaps through hierarchical, tree-based query decomposition. LENS transforms complex evaluation tasks into multi-layered assessment structures via a six-stage pipeline (task formulation, decomposition, tool-augmented execution, structured generation, multi-dimensional scoring, and trace analysis), enabling MRI-like scanning of inference processes to reveal where and why hallucinations originate. The framework introduces four key innovations: (1) Tool Necessity Detection and Selection (TND/TSA) – explicitly evaluating when models should consult external sources versus relying on parametric memory, addressing a fundamental hallucination source. (2) Multi-Dimensional Metrics – assessing degree (accuracy, faithfulness, tool appropriateness), quantity (coverage, completeness), stability (consistency, robustness), and risk (uncertainty quantification) beyond binary detection. (3) User-Centric Benchmark Construction – empowering organizations to design custom evaluations from their evolving knowledge bases while maintaining methodological rigor. (4) Actionable Error Attribution – providing hierarchical decomposition traces with causal attribution, evidence chains, and OpenTelemetry-based reproducibility for transparent auditing. Our systematic taxonomy unifies previously fragmented approaches across evaluation targets (task-specific, modality-based, hallucination-type, domain-specific), dimensions (factuality, faithfulness, consistency, robustness, causal reasoning, interpretability), and methodologies. We introduce unified metrics transcending individual dimensions and present mitigation-aware evaluation strategies integrating RAG, parameter-efficient fine-tuning, knowledge distillation, preference optimization, and temporal intervention approaches. By combining horizontal breadth (across domains and architectures) with vertical depth (into reasoning processes), LENS advances hallucination evaluation from post-hoc error detection to proactive risk assessment. Case studies in medical diagnosis, legal analysis, and financial reasoning demonstrate the framework’s transformative impact, enabling objective model comparison, informed selection, diagnostic insights, domain-evolving benchmarks, and targeted mitigation development—fostering calibrated trust in AI systems deployed in safety-critical applications where accuracy, interpretability, and accountability are indispensable.