When Probes Succeed for the Wrong Reason: Causal Regimes of Probe-Identified Representations
Linear probes are widely used in LLM interpretability, but does high accuracy mean the probe found something causally important? We test this by training probes on refusal behavior in two models, Phi-3 (3.8B) and Mistral-7B, and then using activation interventions to check whether the probe directions are actually necessary. The answer depends on the model. In Phi-3, refusal lives in a low-rank subspace: removing it makes the behavior collapse. In Mistral-7B, refusal is distributed across many redundant directions: adding them steers behavior, but removing them does nothing. High probe accuracy does not tell you which regime you are in.
Authors
- MAITREYA PRASAD WARRIER
- NEHAA SHRI MS (ORCID: https://orcid.org/0009-0001-9304-3039)
Institutions
- Vellore Institute of Technology University (IN)
Publication Details
- Journal
- Zenodo (CERN European Organization for Nuclear Research)
- Published
- 2026-11-16
- DOI
- https://doi.org/10.5281/zenodo.24283763
- Primary Topic
- Explainable Artificial Intelligence (XAI)
- Type
- preprint