Homology Leakage Inflates Performance on a Protein Functional-Class Benchmark
Protein functional-class classifiers are usually evaluated with random train–test splits, which can overstate performance when closely related (homologous) sequences fall in both sets. We re-evaluated a widely used benchmark derived from the Protein Data Bank, with 40,211 unique protein sequences in ten functional classes. Six models were compared under random splits and under identity-controlled splits, which keep sequences sharing more than a set identity (90% down to 20%) on the same side: three k-mer composition classifiers, a method that copies the label of the most similar training sequence found by alignment, and two classifiers built on embeddings from the protein language model ESM-2. Under random splitting, 86.5% of test sequences had a training sequence more than 30% identical to them, and a nearest-neighbour model that only copies labels scored in the same range as the learned models. With related sequences kept apart at 30% identity, the k-mer classifiers lost about two-thirds of their macro-F 1 . As a secondary observation, alignment-based label transfer beat an ESM-2 classifier when close training sequences were available, and ESM-2 did better when they were removed; at 30% identity its macro-F 1 advantage was small and not statistically significant. The reason is unclear: almost every benchmark sequence has a close match in UniRef50, the database used to pretrain ESM-2, leaving too few test sequences distant from both the training set and the pretraining data to test generalization beyond them. Random-split results on this benchmark should not be taken as evidence of functional generalization, and comparisons involving protein language models must also account for pretraining data.
Authors
- Afaque Alam (ORCID: https://orcid.org/0000-0002-5527-651X)
- Mukesh Kumar
Publication Details
- Journal
- Journal of Bioinformatics and Computational Biology
- Published
- 2026-11-19
- DOI
- https://doi.org/10.1142/s0219720026500174
- Primary Topic
- Machine Learning in Bioinformatics
- Type
- article
- Field-Weighted Citation Impact
- 0.00