The Disclosure Effect in LLM Root Cause Analysis: A Truthful Notice About One Injected Fault Suppresses Diagnosis of a Hidden Second Fault

Incident responders are often told which fault a planned fault-injection experiment injected. We ask whether that truthful notice changes whether an LLM responder doing root cause analysis finds a second, independent cause that was not disclosed. The test material is the hybrid (two-fault) subset of OpenRCA 2.0: eighteen cases in which a prominent injected fault and a subtle second fault are present in the same window, both discoverable from the same telemetry, with telemetry byte-identical between experimental arms. Each case is replayed under four prompt conditions — no notice, a neutral notice, a truthful notice naming the injected fault and its target, and the truthful notice plus one counteracting sentence — to three LLM diagnosers, with three repeats per cell: 864 runs. On cases where the hidden fault was found at least once without the notice, the truthful notice reduced hidden-fault recall for one diagnoser by −0.896 (95% CI −1.000 to −0.771, 16 at-risk cases); a second diagnoser moved the same way, inconclusively (−0.156, CI −0.356 to 0.000, 15 cases); a third showed no measurable effect (−0.020, CI −0.176 to +0.098, 17 cases). The neutral notice had no detectable effect for any diagnoser. The counteracting sentence recovered about 60% of the strong diagnoser's loose-score loss and none of its strict-score loss. We claim a per-diagnoser effect, not a general law about LLM responders: telling the responder what was done can stop it looking for what else is wrong, and whether it does depends on the diagnoser.

Authors

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-21
DOI
https://doi.org/10.5281/zenodo.24414406
Primary Topic
Software System Performance and Reliability
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

The Disclosure Effect in LLM Root Cause Analysis: A Truthful Notice About One Injected Fault Suppresses Diagnosis of a Hidden Second Fault

Sajish Krishnan
Zenodo (CERN European Organization for Nuclear Research)
Software System Performance and Reliability
preprint

The Disclosure Effect in LLM Root Cause Analysis: A Truthful Notice About One Injected Fault Suppresses Diagnosis of a Hidden Second Fault

Sajish Krishnan
preprint en

Abstract

Incident responders are often told which fault a planned fault-injection experiment injected. We ask whether that truthful notice changes whether an LLM responder doing root cause analysis finds a second, independent cause that was not disclosed. The test material is the hybrid (two-fault) subset of OpenRCA 2.0: eighteen cases in which a prominent injected fault and a subtle second fault are present in the same window, both discoverable from the same telemetry, with telemetry byte-identical between experimental arms. Each case is replayed under four prompt conditions — no notice, a neutral notice, a truthful notice naming the injected fault and its target, and the truthful notice plus one counteracting sentence — to three LLM diagnosers, with three repeats per cell: 864 runs. On cases where the hidden fault was found at least once without the notice, the truthful notice reduced hidden-fault recall for one diagnoser by −0.896 (95% CI −1.000 to −0.771, 16 at-risk cases); a second diagnoser moved the same way, inconclusively (−0.156, CI −0.356 to 0.000, 15 cases); a third showed no measurable effect (−0.020, CI −0.176 to +0.098, 17 cases). The neutral notice had no detectable effect for any diagnoser. The counteracting sentence recovered about 60% of the strong diagnoser's loose-score loss and none of its strict-score loss. We claim a per-diagnoser effect, not a general law about LLM responders: telling the responder what was done can stop it looking for what else is wrong, and whether it does depends on the diagnoser.

Zenodo (CERN European Organization for Nuclear Research)
Software System Performance and Reliability
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon