ESG Disclosure Quality as Organizational Information Processing: Comparing Human Coding, Rule‐Based Automation, and LLM Semantic Scoring

ABSTRACT This study examines how different scoring architectures evaluate the same ESG disclosure text and produce different ESG disclosure‐quality scores. Using sustainability‐related reports published by Korean listed firms during 2020–2021, the study compares three approaches: Human‐ESG based on a structured nine‐item disclosure‐quality rubric, Rule‐ESG based on a rubric‐aligned deterministic dictionary system, and LLM‐ESG based on semantic scoring through a documented API protocol. All three approaches were applied to the same selected environmental, social, and governance disclosure segments. The analysis evaluates convergent validity, criterion validity, incremental validity, and structural diagnostics using Human‐ESG as a disclosure‐quality benchmark and KCGS ratings as a noisy institutional criterion. The findings indicate that LLM‐ESG shows greater convergence with the human‐coded benchmark and a stronger association with the KCGS institutional criterion than the rule‐based baseline, while producing different distributional properties. Structural diagnostics further suggest that the rule‐based architecture produces stronger upper‐end compression, whereas LLM‐based semantic scoring shows greater cross‐firm differentiation on several diagnostics. Additional robustness and influence diagnostics indicate that the main results are not driven by a small number of influential observations. The study contributes to ESG disclosure research by showing that disagreement in ESG evaluation can arise not only from institutional weighting differences, but also from the scoring architecture used to process disclosure text. The findings further suggest that automated ESG disclosure‐quality scores should be interpreted as architecture‐shaped outputs rather than direct extractions from sustainability reports.