Linear Accessibility Without Sparse Decomposition: Toxin-Related Information in Protein Language Model Representations

A sparse autoencoder trained on ESM-2 650M failed to isolate a compact, stable set of toxin-associated latent features under a preregistered stability criterion. We asked whether that failure reflects an absence of accessible signal or a mismatch between the signal and the autoencoder basis. Using the same frozen discovery universe of 139 animal-toxin proteins and 139 family-aware negatives, we fit L1-regularized logistic probes directly on raw 1,280-dimensional residue-max-pooled ESM-2 representations at six depths, across 100 deterministic perturbations and three per-class sample sizes, with all statistics and mechanics frozen before biological execution. Raw representations carry a strong linearly accessible signal: median held-out AUROC reaches 0.958 at layer 24 against 0.570 for a 21-dimensional sequence-composition baseline, with layer 24 beating its paired baseline in 300/300 perturbation-size comparisons (mean paired ΔAUROC = +0.392). A scoped 100-perturbation label-permutation control at the protocol anchor collapses to chance (median AUROC 0.500, median support recurrence I_stat = 0.000), while the real-label condition reaches 0.931 and I_stat = 0.354, a median separation of +0.431 AUROC. The signal is therefore present and recurrent in the raw basis but did not decompose into the compact stable sparse-autoencoder feature set required by the earlier instrument. We report this as a representation-basis gap and explicitly bound the interpretation: the experiment establishes neither a causal mechanism nor cross-family generalization or a confirmatory verdict. The confirmatory protein universe was never observed.

Authors

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-09-06
DOI
https://doi.org/10.5281/zenodo.22637265
Primary Topic
Machine Learning in Bioinformatics
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
preprint

Linear Accessibility Without Sparse Decomposition: Toxin-Related Information in Protein Language Model Representations

Allan Ochola
Zenodo (CERN European Organization for Nuclear Research)
Machine Learning in Bioinformatics
preprint

Linear Accessibility Without Sparse Decomposition: Toxin-Related Information in Protein Language Model Representations

Allan Ochola
preprint en

Abstract

A sparse autoencoder trained on ESM-2 650M failed to isolate a compact, stable set of toxin-associated latent features under a preregistered stability criterion. We asked whether that failure reflects an absence of accessible signal or a mismatch between the signal and the autoencoder basis. Using the same frozen discovery universe of 139 animal-toxin proteins and 139 family-aware negatives, we fit L1-regularized logistic probes directly on raw 1,280-dimensional residue-max-pooled ESM-2 representations at six depths, across 100 deterministic perturbations and three per-class sample sizes, with all statistics and mechanics frozen before biological execution. Raw representations carry a strong linearly accessible signal: median held-out AUROC reaches 0.958 at layer 24 against 0.570 for a 21-dimensional sequence-composition baseline, with layer 24 beating its paired baseline in 300/300 perturbation-size comparisons (mean paired ΔAUROC = +0.392). A scoped 100-perturbation label-permutation control at the protocol anchor collapses to chance (median AUROC 0.500, median support recurrence I_stat = 0.000), while the real-label condition reaches 0.931 and I_stat = 0.354, a median separation of +0.431 AUROC. The signal is therefore present and recurrent in the raw basis but did not decompose into the compact stable sparse-autoencoder feature set required by the earlier instrument. We report this as a representation-basis gap and explicitly bound the interpretation: the experiment establishes neither a causal mechanism nor cross-family generalization or a confirmatory verdict. The confirmatory protein universe was never observed.

Zenodo (CERN European Organization for Nuclear Research)
Machine Learning in Bioinformatics
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Linear Accessibility Without Sparse Decomposition: Toxin-Related Information in Protein Language Model Representations — Allan Ochola · Zenodo (CERN European Organization for Nuclear Research) (2026) | TGRS Research Map | TGRS