Building a continuous benchmarking ecosystem in bioinformatics
Benchmarking, which involves collecting reference datasets and demonstrating method performance, is a requirement for the development of new computational tools, but also becomes a domain of its own to achieve neutral comparisons of methods. Although a lot has been written about how to design and conduct benchmark studies, this Perspective sheds light on a wish list for a computational platform to orchestrate benchmark studies. We discuss various ideas for organizing reproducible software environments, formally defining benchmarks, orchestrating standardized workflows, and how they interface with computing infrastructure.
Authors
- Charlotte Soneson (ORCID: https://orcid.org/0000-0003-3833-2169)
- Almut Lütge (ORCID: https://orcid.org/0000-0003-1104-727X)
- Izaskun Mallona (ORCID: https://orcid.org/0000-0002-2853-7526)
- Mark D. Robinson (ORCID: https://orcid.org/0000-0002-3048-5518)
- Reto Gerber (ORCID: https://orcid.org/0000-0001-5414-8906)
- Brian Carrillo (ORCID: https://orcid.org/0009-0003-5704-4151)
- Anthony Sonrel (ORCID: https://orcid.org/0000-0002-2414-715X)
- Daniel Incicau (ORCID: https://orcid.org/0009-0001-1748-6145)
Institutions
- SIB Swiss Institute of Bioinformatics (CH)
- University of Zurich (CH)
- Swiss Data Science Center (CH)
- Friedrich Miescher Institute (CH)
Publication Details
- Journal
- PLoS Computational Biology
- Published
- 2025-11-13
- DOI
- https://doi.org/10.1371/journal.pcbi.1013658
- Primary Topic
- Scientific Computing and Data Management
- Type
- article
- Field-Weighted Citation Impact
- 0.00