Are explanations faithful under fire? Evaluating XAI reliability on live honeypot intrusion-detection telemetry

One way to validate explanations from Explainable Artificial Intelligence (XAI) is to use operational data rather than pre-existing datasets. Using this approach, we deployed ten honeypots on Google Cloud Platform for 21 days. During this time, we received 10,900 sessions from 4,817 distinct IP addresses. We manually tagged each session via a human-in-the-loop process (operational annotations; Cohen’s κ = 0.91) and then trained six distinct types of models (Random Forest, XGBoost, LightGBM, MLP, LSTM, and Transformer) with 13 features. Each type of model was evaluated with three protocols – record independent, temporal (hour of day holdout), and source IP group. We evaluated how well each model explained its predictions using frozen checkpoints of a holdout set with two metrics: AUDAC (area under the descriptive-accuracy curve) and flip rate after perturbation. The tree ensembles (Random Forest, XGBoost and LightGBM) had lower AUDAC (indicating more faithful attributions) than the MLP and LSTM models. For instance, LightGBM had an AUDAC of 0.346, XGBoost had an AUDAC of 0.348, and Random Forest had an AUDAC of 0.351. There was consistency among the three seed runs for Random Forest. For LightGBM, the average AUDAC for the three seeds was consistent but the AUDAC for individual seeds varied. For XGBoost, there was no consistency. When we masked the input of the top 10 SHAP features for each model, we found that all correct malicious sessions changed their classification for all three tree-based ensembles (Random Forest, XGBoost and LightGBM). Approximately 98.8 percent of correct malicious sessions were changed for MLP. Under random masking, by contrast, only about 61% of correct malicious sessions flipped. The benign sessions were not flipped by the tree-based ensemble models for Seed 42, and only 2.2% of them were flipped by MLP. Across three seeds, the benign flip rate for Random Forest and LightGBM remained near zero, while the malicious flip rate for XGBoost ranged from 71.8% to 100%. The ranking of SHAP and LIME showed strong correlation (rho = 0.85–0.93) across all five models, although there was less agreement between instances. Due to the transformer’s inability to converge (F1 score of 0.00 on two seeds; produced identical outputs on the third), it was removed from ranking of faithfulness. Ablation studies on service-proxy and Leave One Service Out did not provide any evidence that the tested services relied on service-identity shortcuts. Furthermore, removal of the top three behavioural features used by honeypot signatures (command frequency, authentication failure ratio, and number of unique commands) resulted in relatively small reductions in F1 scores ranging from 3.3% to 3.8% points (approximately 84%–85%). Removal of a fourth feature, suspicious keyword hits, resulted in larger reductions in F1 scores ranging from 4.9–5.7 percentage points. These high levels of performance suggest that the models are using greater amounts of contextual information, rather than simply replicating signature-based rules.