Hierarchical Explanation Graph: A Causal Framework for Explainability in Hierarchical LLM-Based Multi-Agent Systems

Large Language Model (LLM) agents increasingly operate within hierarchical multi-agent pipelines, where decisions propagate through layers of specialized components such as planners, memory systems, executors, and evaluators. When asked to justify their decisions, these agents often generate natural language explanations that appear plausible and confident but may be causally disconnected from the underlying decision-making process, a phenomenon structurally analogous to human confabulation. Existing explainability techniques do not readily generalize to hierarchical agent architectures, in which causal influence is distributed across multiple layers rather than confined to a single inference step. To address this limitation, we introduce the Hierarchical Explanation Graph (HEG), a two-level explainability framework that differentiates between intentional explanations, representing what each layer reports about its own decision process, and causal explanations, representing what controlled perturbation experiments empirically reveal to have influenced the decision. HEG employs a dual-orchestrator architecture for real-time causal intervention: Orchestrator A executes the standard pipeline while recording all intermediate states and layer-level self-explanations, whereas Orchestrator B systematically perturbs intermediate states using a layer-aware Perturbation Engine and propagates these interventions downstream to generate alternative outputs, from which causal weights are estimated through cosine divergence. We evaluate HEG across seven tasks of varying complexity executed within a six-layer pipeline consisting of a Global Planner, Mid-Level Planner, Memory System, Research Agent, Analysis Agent, and Critic Evaluator, performing a total of 42 perturbation experiments. The results demonstrate systematic discrepancies between intentional and causal explanations: the Critic Evaluator exhibits a confabulation rate of 100% despite near-zero causal influence, the Analysis Agent shows a confabulation rate of 43%, while execution-oriented layers such as the Research Agent and Mid-Level Planner display no confabulation. Furthermore, causal weight distributions vary substantially between simple and multi-step tasks, providing empirical evidence that meaningful transparency in hierarchical agent systems requires task-aware, layer-resolved causal attribution rather than reliance on post-hoc narrative explanations.