Lessons learned from the IMMREP23 TCR-epitope prediction challenge

<h2>Abstract</h2> Here, we present the findings from IMMREP23, the second benchmark competition focused on predicting the specificity of TCR-pMHC interactions. The interaction of T cell receptors (TCR) towards their pMHC target is a cornerstone of the cellular immune system. Over the last decade, substantial progress has been made within the field of TCR specificity prediction, providing proof of concept for predicting TCR-pMHC interactions in a narrow space of "seen" pMHC targets where substantial training data is available. However, a significant challenge persists in extending the predictive capability to novel "unseen" pMHC targets. Furthermore, the performance of proposed methods is often challenged when evaluated outside the initial publication and data sets. To address these issues, IMMREP23 challenge invited participants to predict, for a given test set of TCR-pMHC pairs, the likelihood that a pair would bind. A total of 53 teams participated, providing a total of 398 submissions. The benchmark confirms that current methods achieve reasonable performance in the "seen" pMHC setting. However, most participating methods had close to random performance on the subset of "unseen" peptides, underlining that this prediction challenge remains essentially unsolved. Finally, another key lesson from the benchmark is the critical issue of data leakage. Specifically, the data set construction procedure employed in IMMREP23 led to biases in the negative test data set. These biases were identified by several participating teams, and complicated the interpretation of the benchmark results. Based on these results, we put forward suggestions on how future competitions could avoid such data leakages and biases.

Authors

Institutions

Publication Details

Journal
ImmunoInformatics
Published
2024-09-28
DOI
https://doi.org/10.1016/j.immuno.2024.100045
Citations
39
Primary Topic
vaccines and immunoinformatics approaches
Type
article
Field-Weighted Citation Impact
9.12
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
article

Lessons learned from the IMMREP23 TCR-epitope prediction challenge

Sebastiaan Valkiers, Justin Barton, María Rodríguez Martínez, Aleksandra M. Walczak et al.
39 citations
ImmunoInformatics
vaccines and immunoinformatics approaches
9.12
article

Lessons learned from the IMMREP23 TCR-epitope prediction challenge

Sebastiaan Valkiers, Justin Barton, María Rodríguez Martínez, Aleksandra M. Walczak, David Gfeller, Pieter Meysman, Mathias Fynbo Jensen, Dana Léa Moreno, Aurélien Pélissier, Giancarlo Croce, Manisha Goel, Thierry Mora, Morten Nielsen, Anne Eugster, Victor Greiff, Andreas Tiffeau-Mayer, Barthélémy Meynard-Piganeeau
article en
39 citations

Abstract

<h2>Abstract</h2> Here, we present the findings from IMMREP23, the second benchmark competition focused on predicting the specificity of TCR-pMHC interactions. The interaction of T cell receptors (TCR) towards their pMHC target is a cornerstone of the cellular immune system. Over the last decade, substantial progress has been made within the field of TCR specificity prediction, providing proof of concept for predicting TCR-pMHC interactions in a narrow space of "seen" pMHC targets where substantial training data is available. However, a significant challenge persists in extending the predictive capability to novel "unseen" pMHC targets. Furthermore, the performance of proposed methods is often challenged when evaluated outside the initial publication and data sets. To address these issues, IMMREP23 challenge invited participants to predict, for a given test set of TCR-pMHC pairs, the likelihood that a pair would bind. A total of 53 teams participated, providing a total of 398 submissions. The benchmark confirms that current methods achieve reasonable performance in the "seen" pMHC setting. However, most participating methods had close to random performance on the subset of "unseen" peptides, underlining that this prediction challenge remains essentially unsolved. Finally, another key lesson from the benchmark is the critical issue of data leakage. Specifically, the data set construction procedure employed in IMMREP23 led to biases in the negative test data set. These biases were identified by several participating teams, and complicated the interpretation of the benchmark results. Based on these results, we put forward suggestions on how future competitions could avoid such data leakages and biases.

ImmunoInformaticsVol. 16
Oslo University Hospital (NO), Centre National de la Recherche Scientifique (FR), University of Antwerp (BE), University of Oslo (NO), ZHAW Zurich University of Applied Sciences (CH), Université Paris Cité (FR), Université Paris Sciences et Lettres (FR), Centre Hospitalier Universitaire Vaudois (CH), École Normale Supérieure - PSL (FR), Yale University (US), Sorbonne Université (FR), Sorbonne Paris Cité (FR), Biologie Computationnelle, Quantitative et Synthétique (FR), Institut de Biologie Paris-Seine (FR), Ludwig Cancer Research (CH), Institute of Structural and Molecular Biology (GB), University College London (GB), Technische Universität Dresden (DE), ZHAW Zurich University of Applied Sciences (CH), Technical University of Denmark (DK), University of Lausanne (CH), Birkbeck, University of London (GB)
Climate action
Openalex Percentile: Top 1%
vaccines and immunoinformatics approaches
9.12
DOIOpen Access PDFOpenAlex
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.