Homology Leakage Inflates Performance on a Protein Functional-Class Benchmark

Protein functional-class classifiers are usually evaluated with random train–test splits, which can overstate performance when closely related (homologous) sequences fall in both sets. We re-evaluated a widely used benchmark derived from the Protein Data Bank, with 40,211 unique protein sequences in ten functional classes. Six models were compared under random splits and under identity-controlled splits, which keep sequences sharing more than a set identity (90% down to 20%) on the same side: three k-mer composition classifiers, a method that copies the label of the most similar training sequence found by alignment, and two classifiers built on embeddings from the protein language model ESM-2. Under random splitting, 86.5% of test sequences had a training sequence more than 30% identical to them, and a nearest-neighbour model that only copies labels scored in the same range as the learned models. With related sequences kept apart at 30% identity, the k-mer classifiers lost about two-thirds of their macro-F 1 . As a secondary observation, alignment-based label transfer beat an ESM-2 classifier when close training sequences were available, and ESM-2 did better when they were removed; at 30% identity its macro-F 1 advantage was small and not statistically significant. The reason is unclear: almost every benchmark sequence has a close match in UniRef50, the database used to pretrain ESM-2, leaving too few test sequences distant from both the training set and the pretraining data to test generalization beyond them. Random-split results on this benchmark should not be taken as evidence of functional generalization, and comparisons involving protein language models must also account for pretraining data.

Authors

Publication Details

Journal
Journal of Bioinformatics and Computational Biology
Published
2026-11-19
DOI
https://doi.org/10.1142/s0219720026500174
Primary Topic
Machine Learning in Bioinformatics
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

Homology Leakage Inflates Performance on a Protein Functional-Class Benchmark

Afaque Alam, Mukesh Kumar
Journal of Bioinformatics and Computational Biology
Machine Learning in Bioinformatics
article

Homology Leakage Inflates Performance on a Protein Functional-Class Benchmark

Afaque Alam, Mukesh Kumar
article en

Abstract

Protein functional-class classifiers are usually evaluated with random train–test splits, which can overstate performance when closely related (homologous) sequences fall in both sets. We re-evaluated a widely used benchmark derived from the Protein Data Bank, with 40,211 unique protein sequences in ten functional classes. Six models were compared under random splits and under identity-controlled splits, which keep sequences sharing more than a set identity (90% down to 20%) on the same side: three k-mer composition classifiers, a method that copies the label of the most similar training sequence found by alignment, and two classifiers built on embeddings from the protein language model ESM-2. Under random splitting, 86.5% of test sequences had a training sequence more than 30% identical to them, and a nearest-neighbour model that only copies labels scored in the same range as the learned models. With related sequences kept apart at 30% identity, the k-mer classifiers lost about two-thirds of their macro-F 1 . As a secondary observation, alignment-based label transfer beat an ESM-2 classifier when close training sequences were available, and ESM-2 did better when they were removed; at 30% identity its macro-F 1 advantage was small and not statistically significant. The reason is unclear: almost every benchmark sequence has a close match in UniRef50, the database used to pretrain ESM-2, leaving too few test sequences distant from both the training set and the pretraining data to test generalization beyond them. Random-split results on this benchmark should not be taken as evidence of functional generalization, and comparisons involving protein language models must also account for pretraining data.

Journal of Bioinformatics and Computational Biology
Machine Learning in Bioinformatics
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon