Post-hoc Interpretability of LDA on Hyperspectral Imagery: SHAP Attributions, Counterfactual Topic Flips, and LLM-judge Alignment under Token-Mass-Dispersion Asymmetry

Programme paper P5 of the CAOS_LDA_HSI series. The interpretability claim of LDA-on-HSI rests on the assumption that topic-word distributions are human-readable. This study is restricted to the LDA backbone; the cross-backbone comparison (HDP, ProdLDA, ETM) is the subject of the companion paper P4. We test three orthogonal post-hoc interpretability axes on the V1-V15, V17-V20 wordification sweep (nineteen LDA-fitted recipes) from companion paper P3: F-13 SHAP attribution of pixel-to-topic decisions via a closed-form posterior, F-22 counterfactual L1 perturbation required to flip a document's argmax topic, and F-15 LLM-as-judge alignment between a document's top tokens and its argmax topic's top tokens. Three concrete results. (1) F-13 SHAP gives recipe-specific explanations consistent across scenes: V1 reduces to specific wavelength bands, V7 to absorption features, V12 to clusters of Gaussian-mixture components. (2) F-22 counterfactual L1 separates recipes into robust (V20=26.3, V12=24.5, V3=23.5), moderate (V1=6.1, V7=5.2) and fragile (V9=1.0, V10=1.2) regimes; the most-robust basis (V20) is ~26x harder to flip than V9's. (3) F-15 LLM-judge alignment is anti-correlated with F-2 coherence on large-vocabulary recipes, but cardinality cannot explain the gap: V20 matches V3/V12 alphabet sizes yet scores F-15 = 0.64 against their 0.12/0.16. We make the mechanism quantitative with the token-mass dispersion metric (effective vocabulary N_eff = exp(H(phi_k))): N_eff falls V3 (431) > V12 (400) > V20 (309) while F-15 rises in lock-step, so the artefact is driven by token-mass dispersion, not nominal vocabulary cardinality. We recommend reporting F-13 SHAP top-K attributions as the primary interpretability artefact, F-22 as the topic-stability artefact, and F-15 only after dispersion normalisation. Code and derived artefacts: https://github.com/fsantibanezleal/CAOS_LDA_HSI . Interactive web application: https://lda-hsi.fasl-work.com . Manuscript sources: https://github.com/fsantibanezleal/CAOS_LDA_HSI_Paper . Funding: The Advanced Mining Technology Center (AMTC) Basal project (ANID/PIA Project AFB220002) and ANID FONDECYT Postdoctorado 3220094.

Authors

Institutions

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-07-23
DOI
https://doi.org/10.5281/zenodo.21504114
Primary Topic
Explainable Artificial Intelligence (XAI)
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
preprint

Post-hoc Interpretability of LDA on Hyperspectral Imagery: SHAP Attributions, Counterfactual Topic Flips, and LLM-judge Alignment under Token-Mass-Dispersion Asymmetry

Felipe A. Santibáñez-Leal
Zenodo (CERN European Organization for Nuclear Research)
Explainable Artificial Intelligence (XAI)
preprint

Post-hoc Interpretability of LDA on Hyperspectral Imagery: SHAP Attributions, Counterfactual Topic Flips, and LLM-judge Alignment under Token-Mass-Dispersion Asymmetry

Felipe A. Santibáñez-Leal
preprint en

Abstract

Programme paper P5 of the CAOS_LDA_HSI series. The interpretability claim of LDA-on-HSI rests on the assumption that topic-word distributions are human-readable. This study is restricted to the LDA backbone; the cross-backbone comparison (HDP, ProdLDA, ETM) is the subject of the companion paper P4. We test three orthogonal post-hoc interpretability axes on the V1-V15, V17-V20 wordification sweep (nineteen LDA-fitted recipes) from companion paper P3: F-13 SHAP attribution of pixel-to-topic decisions via a closed-form posterior, F-22 counterfactual L1 perturbation required to flip a document's argmax topic, and F-15 LLM-as-judge alignment between a document's top tokens and its argmax topic's top tokens. Three concrete results. (1) F-13 SHAP gives recipe-specific explanations consistent across scenes: V1 reduces to specific wavelength bands, V7 to absorption features, V12 to clusters of Gaussian-mixture components. (2) F-22 counterfactual L1 separates recipes into robust (V20=26.3, V12=24.5, V3=23.5), moderate (V1=6.1, V7=5.2) and fragile (V9=1.0, V10=1.2) regimes; the most-robust basis (V20) is ~26x harder to flip than V9's. (3) F-15 LLM-judge alignment is anti-correlated with F-2 coherence on large-vocabulary recipes, but cardinality cannot explain the gap: V20 matches V3/V12 alphabet sizes yet scores F-15 = 0.64 against their 0.12/0.16. We make the mechanism quantitative with the token-mass dispersion metric (effective vocabulary N_eff = exp(H(phi_k))): N_eff falls V3 (431) > V12 (400) > V20 (309) while F-15 rises in lock-step, so the artefact is driven by token-mass dispersion, not nominal vocabulary cardinality. We recommend reporting F-13 SHAP top-K attributions as the primary interpretability artefact, F-22 as the topic-stability artefact, and F-15 only after dispersion normalisation. Code and derived artefacts: https://github.com/fsantibanezleal/CAOS_LDA_HSI . Interactive web application: https://lda-hsi.fasl-work.com . Manuscript sources: https://github.com/fsantibanezleal/CAOS_LDA_HSI_Paper . Funding: The Advanced Mining Technology Center (AMTC) Basal project (ANID/PIA Project AFB220002) and ANID FONDECYT Postdoctorado 3220094.

Zenodo (CERN European Organization for Nuclear Research)
Open University of Cyprus (CY)
Explainable Artificial Intelligence (XAI)
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.