Does your readout see the input? A sanity check for linear probing of representations

To ask what a representation contains, researchers usually fit a linear readout: a regression or classifier trained on it. That readout is a model with inputs of its own. It can be given the representation alone, or together with the original features: two different questions, two different answers. Given both, its score stops measuring the representation and starts measuring the input. On a public fMRI dataset the arithmetic reproduces to the last digit, and the same design raises reported decoding accuracy by 22% to 326%. I show that this is not a bias of unknown size but a quantity that can be computed from two scores. The apparent advantage of supplying the features equals C/B − 1, where B is the score of a readout fitted on the representation alone and C the score of one fitted on the features alone. In a reconstruction probe the features-only readout became numerically indistinguishable from a readout given the full representation, so the metric lost its ability to rank representations at all. With a classifier, the same setting inflated accuracy by tens of percent. The effect tracked how much of the input the representation retained and did not depend on how the representation was learned: a trained autoencoder reproduced the result of PCA, and a second dataset reproduced it again. I give a seven-item checklist and a one-function diagnostic built only on scikit-learn, and I report the predictions I wrote down before running, including the two that my data refuted.

Authors

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-13
DOI
https://doi.org/10.5281/zenodo.24183421
Primary Topic
Explainable Artificial Intelligence (XAI)
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Does your readout see the input? A sanity check for linear probing of representations

Yuntong Jiang
Zenodo (CERN European Organization for Nuclear Research)
Explainable Artificial Intelligence (XAI)
preprint

Does your readout see the input? A sanity check for linear probing of representations

Yuntong Jiang
preprint en

Abstract

To ask what a representation contains, researchers usually fit a linear readout: a regression or classifier trained on it. That readout is a model with inputs of its own. It can be given the representation alone, or together with the original features: two different questions, two different answers. Given both, its score stops measuring the representation and starts measuring the input. On a public fMRI dataset the arithmetic reproduces to the last digit, and the same design raises reported decoding accuracy by 22% to 326%. I show that this is not a bias of unknown size but a quantity that can be computed from two scores. The apparent advantage of supplying the features equals C/B − 1, where B is the score of a readout fitted on the representation alone and C the score of one fitted on the features alone. In a reconstruction probe the features-only readout became numerically indistinguishable from a readout given the full representation, so the metric lost its ability to rank representations at all. With a classifier, the same setting inflated accuracy by tens of percent. The effect tracked how much of the input the representation retained and did not depend on how the representation was learned: a trained autoencoder reproduced the result of PCA, and a second dataset reproduced it again. I give a seven-item checklist and a one-function diagnostic built only on scikit-learn, and I report the predictions I wrote down before running, including the two that my data refuted.

Zenodo (CERN European Organization for Nuclear Research)
Explainable Artificial Intelligence (XAI)
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon