Using a continuous benchmarking platform to compare different variant calling pipelines across Germany

Famke Bäuerle1,2, Bianca K. Stöcker3, Winfried Hofmann4, Michael Menzel5, Aurelie Tomczak5, Jan Budczies5, Andreas Blaumeiser6, Thomas Pauli6, Vincent Schipperges6, Patrick Metzger6, Sabrina Krakau2, Sven Nahnsen2, Oliver Kohlbacher1 and Johannes Köster3 We conducted a comparative benchmark of different variant calling pipelines for clinical sequencing in German university hospitals as part of the nationwide PM4Onco project, with the objective of implementing personalised medicine for cancer patients. The process of variant calling takes sequencing data as input and reports genetic variants as the output. A pipeline will concatenate all tools necessary to pre-process the sequencing data, conduct the variant calling itself, apply certain filters or annotate the variants. Many parameters can be tweaked when developing such a pipeline. Two main applications are available for variant calling: the identification of germline variants or, in our case, the identification of somatic variants. Somatic variants can be determined through tumor-normal calling, in which variants from the healthy tissue are compared to variants identified in tumor tissue, and only those variants associated with the tumor tissue are kept. While various suitable datasets are available for benchmarking germline variant calling, only a few datasets are currently available for benchmarking somatic variant calling. We used two different datasets each associated with drawbacks that we had to address during analysis. We used an Open Science Framework (OSF) project to allow uploads of pipeline results and provided a YAML file to be completed by the participants. This method helped us with collecting metadata about the different pipelines and enabled the downstream comparison. The continuous benchmarking platform NCBench, as described by Hanssen et al., was extended to allow benchmarking of somatic variant calling pipelines. We used this platform to evaluate the files coming from the different variant calling pipelines. Given the continuous development of tools, reference data and pipeline implementations, benchmarking those developments in real-time is desired. NCBench facilitates this process by enabling the developer to run their local pipeline and upload the generated VCFs to Zenodo and then create a pull request on the NCBench GitHub repository. We will present the results of a variant-calling pipeline benchmark across locations in Germany with a focus on discussing suitable datasets, organizational aspects and the continuous benchmarking platform NCBench. 1 Institute for Translational Bioinformatics, University Hospital Tübingen, Tübingen, Germany 2 Quantitative Biology Center, Eberhard-Karls University of Tübingen, Otfried-Müller Str. 37, Tübingen 72076, Baden-Württemberg, Germany. 3 Bioinformatics and Computational Oncology, Institute for AI in Medicine (IKIM), University Hospital Essen, Faculty of Medicine, University of Duisburg-Essen 4 Department of Human Genetics, Hannover Medical School, Hannover, Germany 5 Institute of Pathology, Heidelberg University Hospital, Heidelberg, Germany 6 Institute of Medical Bioinformatics and Systems Medicine, Medical Center-University of Freiburg, Faculty of Medicine, University of Freiburg, Freiburg, Germany