Critical assessment of missense variant effect predictors on disease-relevant variant data

Regular, systematic, and independent assessments of computational tools that are used to predict the pathogenicity of missense variants are necessary to evaluate their clinical and research utility and guide future improvements. The Critical Assessment of Genome Interpretation (CAGI) conducts the ongoing Annotate-All-Missense (Missense Marathon) challenge, in which missense variant effect predictors (also called variant impact predictors) are evaluated on missense variants added to disease-relevant databases following the prediction submission deadline. Here we assess predictors submitted to the CAGI 6 Annotate-All-Missense challenge, predictors commonly used in clinical genetics, and recently developed deep learning methods. We examine performance across a range of settings relevant for clinical and research applications, focusing on different subsets of the evaluation data as well as high-specificity and high-sensitivity regimes. Our evaluations reveal notable advances in current methods relative to older, well-cited tools in the field. While meta-predictors tend to outperform their constituent individual predictors, several newer individual predictors perform comparably to commonly used meta-predictors. Predictor performance varies between high-specificity and high-sensitivity regimes, highlighting that different methods may be optimal for different use cases. We also characterize two potential sources of bias. Predictors that incorporate allele frequency as a predictive feature tend to have reduced performance when distinguishing pathogenic variants from very rare benign variants, and predictors trained on pathogenicity labels from curated variant databases often inherit gene-level label imbalances. Our findings help illuminate the clinical and research utility of modern missense variant effect predictors and identify potential areas for future development.

Authors

Institutions

Publication Details

Journal
Human Genetics
Published
2025-03-01
DOI
https://doi.org/10.1007/s00439-025-02732-2
Citations
18
Primary Topic
Genomics and Rare Diseases
Type
article
Field-Weighted Citation Impact
32.71

Funders

Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
article

Critical assessment of missense variant effect predictors on disease-relevant variant data

Tobias Olenyi, François Ancien, Céline Marquet, Constantina Bakolitsa et al.
18 citations
Human Genetics
Genomics and Rare Diseases
32.71
article

Critical assessment of missense variant effect predictors on disease-relevant variant data

Tobias Olenyi, François Ancien, Céline Marquet, Constantina Bakolitsa, Daniele Raimondi, Changwon Keum, Olivier Poch, Kirsley Chennen, Yisu Peng, Sindy Li, Matthew Mort, D.N. Cooper, Giulia Babbi, Pier Luigi Martelli, Steven E. Brenner, Wim Vranken, Predrag Radivojac, Rita Casadio, Nilah M. Ioannidis, Burkhard Rost, Junwoo Woo, Castrense Savojardo, Ryan Chung, Marianne Rooman, Gabriel Cia, Chang Li, Thomas Weber, Ruchir Rastogi, Vikas Pejaver, Fabrizio Pucci, Kyoungyeul Lee, Timothy Bergquist, Gaia Andreoletti, Akash Kamandula, Dongwook Kim, Xiaoming Liu
article en
18 citations

Abstract

Regular, systematic, and independent assessments of computational tools that are used to predict the pathogenicity of missense variants are necessary to evaluate their clinical and research utility and guide future improvements. The Critical Assessment of Genome Interpretation (CAGI) conducts the ongoing Annotate-All-Missense (Missense Marathon) challenge, in which missense variant effect predictors (also called variant impact predictors) are evaluated on missense variants added to disease-relevant databases following the prediction submission deadline. Here we assess predictors submitted to the CAGI 6 Annotate-All-Missense challenge, predictors commonly used in clinical genetics, and recently developed deep learning methods. We examine performance across a range of settings relevant for clinical and research applications, focusing on different subsets of the evaluation data as well as high-specificity and high-sensitivity regimes. Our evaluations reveal notable advances in current methods relative to older, well-cited tools in the field. While meta-predictors tend to outperform their constituent individual predictors, several newer individual predictors perform comparably to commonly used meta-predictors. Predictor performance varies between high-specificity and high-sensitivity regimes, highlighting that different methods may be optimal for different use cases. We also characterize two potential sources of bias. Predictors that incorporate allele frequency as a predictive feature tend to have reduced performance when distinguishing pathogenic variants from very rare benign variants, and predictors trained on pathogenicity labels from curated variant databases often inherit gene-level label imbalances. Our findings help illuminate the clinical and research utility of modern missense variant effect predictors and identify potential areas for future development.

Human GeneticsVol. 144(2-3)
Northeastern University (US), Université Libre de Bruxelles (BE), Sage Bionetworks (US), Vrije Universiteit Brussel (BE), Université de Montpellier (FR), University of South Florida (US), Institut de Génétique Moléculaire de Montpellier (FR), Chan Zuckerberg Initiative (United States) (US), Genomic Health (United States) (US), Technical University of Munich (DE), Université de Strasbourg (FR), Cardiff University (GB), University of Bologna (IT), University of California, Berkeley (US), Icahn School of Medicine at Mount Sinai (US), KU Leuven (BE)
National Institutes of Health
Openalex Percentile: Top 0%
Genomics and Rare Diseases
32.71
DOIOpen Access PDFOpenAlex
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.