Linear Accessibility Without Sparse Decomposition: Toxin-Related Information in Protein Language Model Representations
A sparse autoencoder trained on ESM-2 650M failed to isolate a compact, stable set of toxin-associated latent features under a preregistered stability criterion. We asked whether that failure reflects an absence of accessible signal or a mismatch between the signal and the autoencoder basis. Using the same frozen discovery universe of 139 animal-toxin proteins and 139 family-aware negatives, we fit L1-regularized logistic probes directly on raw 1,280-dimensional residue-max-pooled ESM-2 representations at six depths, across 100 deterministic perturbations and three per-class sample sizes, with all statistics and mechanics frozen before biological execution. Raw representations carry a strong linearly accessible signal: median held-out AUROC reaches 0.958 at layer 24 against 0.570 for a 21-dimensional sequence-composition baseline, with layer 24 beating its paired baseline in 300/300 perturbation-size comparisons (mean paired ΔAUROC = +0.392). A scoped 100-perturbation label-permutation control at the protocol anchor collapses to chance (median AUROC 0.500, median support recurrence I_stat = 0.000), while the real-label condition reaches 0.931 and I_stat = 0.354, a median separation of +0.431 AUROC. The signal is therefore present and recurrent in the raw basis but did not decompose into the compact stable sparse-autoencoder feature set required by the earlier instrument. We report this as a representation-basis gap and explicitly bound the interpretation: the experiment establishes neither a causal mechanism nor cross-family generalization or a confirmatory verdict. The confirmatory protein universe was never observed.
Authors
- Allan Ochola (ORCID: https://orcid.org/0000-0002-0480-0123)
Publication Details
- Journal
- Zenodo (CERN European Organization for Nuclear Research)
- Published
- 2026-09-06
- DOI
- https://doi.org/10.5281/zenodo.22637265
- Primary Topic
- Machine Learning in Bioinformatics
- Type
- preprint