KoDocXAI: A Quantitative Evaluation Framework and Benchmark for Explainability in Key Information Extraction from Korean Administrative Form Documents

Key information extraction (KIE) models are increasingly used in high-stakes document workflows, where each extracted value must be accompanied by evidence a reviewer can verify. Evaluating their explanations is difficult: human rationale annotation is costly, and in agglutinative Korean, subword tokens misalign with the units humans read. We present KoDocXAI, an evaluation framework and pilot benchmark for the explanations of KIE models on Korean administrative form documents. It derives a value-region rationale reference from existing KIE bounding-box annotations and scores value-region plausibility, whether or not an explanation localizes the extracted value; re-aggregates subword attributions to morphemes and eojeols (Korean spacing units); and measures faithfulness by modality-wise perturbation with the pretraining [MASK] token while protecting the ground-truth span. The pilot benchmark v1.0 covers five fields over 300 permit-and-authorization dispatch documents, 20% of which are doubly annotated. With a fine-tuned LayoutXLM, Integrated Gradients exceeds a random baseline on both axes, whereas attention rollout stays at baseline level under both a first-token and a span-level target. Span protection removes inflated faithfulness scores and a spurious aggregation-level effect. The modality interventions differed in strength, but under them the image channel contributed practically nothing, confining the validated scope to text and layout. These results hold for this document type and model; they support validating the measurement protocol itself and reporting explainability separately by axis and modality.