The Disclosure Effect in LLM Root Cause Analysis: A Truthful Notice About One Injected Fault Suppresses Diagnosis of a Hidden Second Fault
Incident responders are often told which fault a planned fault-injection experiment injected. We ask whether that truthful notice changes whether an LLM responder doing root cause analysis finds a second, independent cause that was not disclosed. The test material is the hybrid (two-fault) subset of OpenRCA 2.0: eighteen cases in which a prominent injected fault and a subtle second fault are present in the same window, both discoverable from the same telemetry, with telemetry byte-identical between experimental arms. Each case is replayed under four prompt conditions — no notice, a neutral notice, a truthful notice naming the injected fault and its target, and the truthful notice plus one counteracting sentence — to three LLM diagnosers, with three repeats per cell: 864 runs. On cases where the hidden fault was found at least once without the notice, the truthful notice reduced hidden-fault recall for one diagnoser by −0.896 (95% CI −1.000 to −0.771, 16 at-risk cases); a second diagnoser moved the same way, inconclusively (−0.156, CI −0.356 to 0.000, 15 cases); a third showed no measurable effect (−0.020, CI −0.176 to +0.098, 17 cases). The neutral notice had no detectable effect for any diagnoser. The counteracting sentence recovered about 60% of the strong diagnoser's loose-score loss and none of its strict-score loss. We claim a per-diagnoser effect, not a general law about LLM responders: telling the responder what was done can stop it looking for what else is wrong, and whether it does depends on the diagnoser.
Authors
- Sajish Krishnan (ORCID: https://orcid.org/0009-0009-7561-9290)
Publication Details
- Journal
- Zenodo (CERN European Organization for Nuclear Research)
- Published
- 2026-11-21
- DOI
- https://doi.org/10.5281/zenodo.24414406
- Primary Topic
- Software System Performance and Reliability
- Type
- preprint