What enzyme-function scores hide: an audit of CLEAN with pre-registered tests

Abstract Enzyme Commission (EC) number predictors such as CLEAN are evaluated with support-weighted precision and F1, which ignore the precision loss from predicted EC numbers absent from the test labels. This scoring, used in the public code of CLEAN, three subsequent methods and the EC-Bench bench- mark, overlooks 67–96% of CLEAN’s false positives on standard benchmarks, mostly incorrect fourth digits. Rescoring EC-Bench predictions for 16 meth- ods showed score inflation rising with the number of EC numbers assigned and one significant pair reversal, while overall rankings remained similar (Kendall’s τb = 0.90). In a temporal benchmark of 2,099 Swiss-Prot enzymes added after CLEAN’s training snapshot, 30.5% carried EC numbers unseen during train- ing. For the remaining enzymes, inference from neighboring training members in CLEAN’s embedding space outperformed its centroid rule (pre-registered: union-label macro-F1 +0.043), although the class-size correction added noth- ing. In descriptive analyses, BLASTp outperformed every embedding-based rule (micro-F1 0.735 vs 0.641 for the centroid rule and 0.703 for the best alternative, exemplar), with no significant difference from the centroid below 30% sequence identity. A subgroup analysis registered afterwards showed a BLASTp advan- tage on experimentally supported annotations (+0.082). CLEAN’s confidence exceeded its accuracy by 0.16–0.30 for rare EC numbers, and model-based cali- bration budgets for 20% rare-tier error had lower bootstrap limits 4.5–6.7 times the available sample sizes. We propose an EC evaluation card for comparable, transparent reporting.

Authors

Institutions

Publication Details

Journal
Research Square
Published
2026-11-17
DOI
https://doi.org/10.21203/rs.3.rs-11732908/v1
Primary Topic
Machine Learning in Bioinformatics
Type
preprint

Funders

Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

What enzyme-function scores hide: an audit of CLEAN with pre-registered tests

Prateek Paudel, Prativa Khanal
2 links
Research Square
Machine Learning in Bioinformatics
preprint

What enzyme-function scores hide: an audit of CLEAN with pre-registered tests

Prateek Paudel, Prativa Khanal
preprint en

Abstract

Abstract Enzyme Commission (EC) number predictors such as CLEAN are evaluated with support-weighted precision and F1, which ignore the precision loss from predicted EC numbers absent from the test labels. This scoring, used in the public code of CLEAN, three subsequent methods and the EC-Bench bench- mark, overlooks 67–96% of CLEAN’s false positives on standard benchmarks, mostly incorrect fourth digits. Rescoring EC-Bench predictions for 16 meth- ods showed score inflation rising with the number of EC numbers assigned and one significant pair reversal, while overall rankings remained similar (Kendall’s τb = 0.90). In a temporal benchmark of 2,099 Swiss-Prot enzymes added after CLEAN’s training snapshot, 30.5% carried EC numbers unseen during train- ing. For the remaining enzymes, inference from neighboring training members in CLEAN’s embedding space outperformed its centroid rule (pre-registered: union-label macro-F1 +0.043), although the class-size correction added noth- ing. In descriptive analyses, BLASTp outperformed every embedding-based rule (micro-F1 0.735 vs 0.641 for the centroid rule and 0.703 for the best alternative, exemplar), with no significant difference from the centroid below 30% sequence identity. A subgroup analysis registered afterwards showed a BLASTp advan- tage on experimentally supported annotations (+0.082). CLEAN’s confidence exceeded its accuracy by 0.16–0.30 for rare EC numbers, and model-based cali- bration budgets for 20% rare-tier error had lower bootstrap limits 4.5–6.7 times the available sample sizes. We propose an EC evaluation card for comparable, transparent reporting.

Research Square
Kennesaw State University (US)
University of Illinois at Urbana-Champaign
Machine Learning in Bioinformatics
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon