When Probes Succeed for the Wrong Reason: Causal Regimes of Probe-Identified Representations

Linear probes are widely used in LLM interpretability, but does high accuracy mean the probe found something causally important? We test this by training probes on refusal behavior in two models, Phi-3 (3.8B) and Mistral-7B, and then using activation interventions to check whether the probe directions are actually necessary. The answer depends on the model. In Phi-3, refusal lives in a low-rank subspace: removing it makes the behavior collapse. In Mistral-7B, refusal is distributed across many redundant directions: adding them steers behavior, but removing them does nothing. High probe accuracy does not tell you which regime you are in.

Authors

Institutions

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-16
DOI
https://doi.org/10.5281/zenodo.24283763
Primary Topic
Explainable Artificial Intelligence (XAI)
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

When Probes Succeed for the Wrong Reason: Causal Regimes of Probe-Identified Representations

MAITREYA PRASAD WARRIER, NEHAA SHRI MS
Zenodo (CERN European Organization for Nuclear Research)
Explainable Artificial Intelligence (XAI)
preprint

When Probes Succeed for the Wrong Reason: Causal Regimes of Probe-Identified Representations

MAITREYA PRASAD WARRIER, NEHAA SHRI MS
preprint en

Abstract

Linear probes are widely used in LLM interpretability, but does high accuracy mean the probe found something causally important? We test this by training probes on refusal behavior in two models, Phi-3 (3.8B) and Mistral-7B, and then using activation interventions to check whether the probe directions are actually necessary. The answer depends on the model. In Phi-3, refusal lives in a low-rank subspace: removing it makes the behavior collapse. In Mistral-7B, refusal is distributed across many redundant directions: adding them steers behavior, but removing them does nothing. High probe accuracy does not tell you which regime you are in.

Zenodo (CERN European Organization for Nuclear Research)
Vellore Institute of Technology University (IN)
Explainable Artificial Intelligence (XAI)
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon