Curated annotations locate biological concepts in protein language models from layers to neurons

Interpretability methods describe what a network has learned, yet their claims are rarely tested against model-independent ground truth. We argue that proteins are an ideal modality for interpretability since curators annotate functional sites residue by residue. Across three ESMC protein language models (300M-6B parameters), we show that conclusions depend on where models are read, how concepts are decoded, which units are examined, and how they are scored. Concepts become linearly readable along depth long before erasure impairs masked-token prediction, and 20-fold scaling shifts concept depth more than peak score. Most of a nonlinear probe's lead over a linear one reflects fitting and readout shape rather than nonlinear encoding. Over whole proteins, saturated metrics render feed-forward neurons indistinguishable from random rotations. At annotated residues, dual-signed neuron poles identify 2.5-5.4 times as many concepts as rotations without training, though sparse autoencoders identify more. Without held-out validation, however, shuffled annotations label nearly every latent, and causal reliance emerges in neuron groups rather than single poles. Reading selected intra-block states exposes binding and active sites that stream probes read poorly and lifts a simple probe 5% above the best published MCC on a leakage-controlled interaction benchmark. We recommend reading a model directly against annotation, scoring units where they fire with a measure that is not saturated, and training a dictionary only where it adds coverage.

Authors

Institutions

Publication Details

Journal
bioRxiv (Cold Spring Harbor Laboratory)
Published
2026-11-20
DOI
https://doi.org/10.64898/2026.11.19.767884
Primary Topic
Explainable Artificial Intelligence (XAI)
Type
preprint

Funders

Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Curated annotations locate biological concepts in protein language models from layers to neurons

Jason P. Gleghorn, Logan Hallee
1 links
bioRxiv (Cold Spring Harbor Laboratory)
Explainable Artificial Intelligence (XAI)
preprint

Curated annotations locate biological concepts in protein language models from layers to neurons

Jason P. Gleghorn, Logan Hallee
preprint en

Abstract

Interpretability methods describe what a network has learned, yet their claims are rarely tested against model-independent ground truth. We argue that proteins are an ideal modality for interpretability since curators annotate functional sites residue by residue. Across three ESMC protein language models (300M-6B parameters), we show that conclusions depend on where models are read, how concepts are decoded, which units are examined, and how they are scored. Concepts become linearly readable along depth long before erasure impairs masked-token prediction, and 20-fold scaling shifts concept depth more than peak score. Most of a nonlinear probe's lead over a linear one reflects fitting and readout shape rather than nonlinear encoding. Over whole proteins, saturated metrics render feed-forward neurons indistinguishable from random rotations. At annotated residues, dual-signed neuron poles identify 2.5-5.4 times as many concepts as rotations without training, though sparse autoencoders identify more. Without held-out validation, however, shuffled annotations label nearly every latent, and causal reliance emerges in neuron groups rather than single poles. Reading selected intra-block states exposes binding and active sites that stream probes read poorly and lifts a simple probe 5% above the best published MCC on a leakage-controlled interaction benchmark. We recommend reading a model directly against annotation, scoring units where they fire with a measure that is not saturated, and training a dictionary only where it adds coverage.

bioRxiv (Cold Spring Harbor Laboratory)
University of Delaware (US)
National Science Foundation, University of Delaware, National Institutes of Health, National Institute of General Medical Sciences
Explainable Artificial Intelligence (XAI)
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon
Curated annotations locate biological concepts in protein language models from layers to neurons — Jason P. Gleghorn, Logan Hallee · bioRxiv (Cold Spring Harbor Laboratory) (2026) | TGRS Research Map | TGRS