Multi-modal dataset creation for federated learning with DICOM-structured reports

Purpose Federated training is often challenging on heterogeneous datasets due to divergent data storage options, inconsistent naming schemes, varied annotation procedures, and disparities in label quality. This is particularly evident in the emerging multi-modal learning paradigms, where dataset harmonization including a uniform data representation and filtering options are of paramount importance.Methods DICOM-structured reports enable the standardized linkage of arbitrary information beyond the imaging domain and can be used within Python deep learning pipelines with highdicom. Building on this, we developed an open platform for data integration with interactive filtering capabilities, thereby simplifying the process of creation of patient cohorts over several sites with consistent multi-modal data.Results In this study, we extend our prior work by showing its applicability to more and divergent data types, as well as streamlining datasets for federated training within an established consortium of eight university hospitals in Germany. We prove its concurrent filtering ability by creating harmonized multi-modal datasets across all locations for predicting the outcome after minimally invasive heart valve replacement. The data include imaging and waveform data (i.e., computed tomography images, electrocardiography scans) as well as annotations (i.e., calcification segmentations, and pointsets), and metadata (i.e., prostheses and pacemaker dependency).Conclusion Structured reports bridge the traditional gap between imaging systems and information systems. Utilizing the inherent DICOM reference system arbitrary data types can be queried concurrently to create meaningful cohorts for multi-centric data analysis. The graphical interface as well as example structured report templates are available at https://github.com/Cardio-AI/fl-multi-modal-dataset-creation .

Authors

Institutions

Publication Details

Journal
International Journal of Computer Assisted Radiology and Surgery
Published
2025-02-03
DOI
https://doi.org/10.1007/s11548-025-03327-y
Citations
6
Primary Topic
Privacy-Preserving Technologies in Data
Type
article
Field-Weighted Citation Impact
14.89

Funders

Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
article

Multi-modal dataset creation for federated learning with DICOM-structured reports

Malte Tölle, Norbert Frey, Clemens Scherer, Stefan Simm et al.
6 citations
International Journal of Computer Assisted Radiology and Surgery
Privacy-Preserving Technologies in Data
14.89
article

Multi-modal dataset creation for federated learning with DICOM-structured reports

Malte Tölle, Norbert Frey, Clemens Scherer, Stefan Simm, Anja Hennemuth, Andreas Leha, Gerhard‐Paul Diller, Peter Bannas, Tim Seidler, Stefan Groß, Stefan Orwat, Lars Kaderali, Nina Krüger, Tim Friede, Halvar Kelm, Lukas Burger, Moritz Seiffert, Eike Nagel, Alexander Meyer, Jan Moritz Seliger, Sandy Engelhardt, Simon S. Martin, Florian André, Philipp Garthe
article en
6 citations

Abstract

Purpose Federated training is often challenging on heterogeneous datasets due to divergent data storage options, inconsistent naming schemes, varied annotation procedures, and disparities in label quality. This is particularly evident in the emerging multi-modal learning paradigms, where dataset harmonization including a uniform data representation and filtering options are of paramount importance.Methods DICOM-structured reports enable the standardized linkage of arbitrary information beyond the imaging domain and can be used within Python deep learning pipelines with highdicom. Building on this, we developed an open platform for data integration with interactive filtering capabilities, thereby simplifying the process of creation of patient cohorts over several sites with consistent multi-modal data.Results In this study, we extend our prior work by showing its applicability to more and divergent data types, as well as streamlining datasets for federated training within an established consortium of eight university hospitals in Germany. We prove its concurrent filtering ability by creating harmonized multi-modal datasets across all locations for predicting the outcome after minimally invasive heart valve replacement. The data include imaging and waveform data (i.e., computed tomography images, electrocardiography scans) as well as annotations (i.e., calcification segmentations, and pointsets), and metadata (i.e., prostheses and pacemaker dependency).Conclusion Structured reports bridge the traditional gap between imaging systems and information systems. Utilizing the inherent DICOM reference system arbitrary data types can be queried concurrently to create meaningful cohorts for multi-centric data analysis. The graphical interface as well as example structured report templates are available at https://github.com/Cardio-AI/fl-multi-modal-dataset-creation .

International Journal of Computer Assisted Radiology and SurgeryVol. 20(3)
Goethe University Frankfurt (DE), Universität Hamburg (DE), Zimmer Biomet (Netherlands) (NL), Heidelberg University (DE), Universitätsmedizin Greifswald (DE), University Hospital Heidelberg (DE), LMU Klinikum (DE), Humboldt-Universität zu Berlin (DE), University Hospital Münster (DE), Deutsches Herzzentrum der Charité (DE), University Medical Center Hamburg-Eppendorf (DE), German Center for Infection Research (DE), Universitätsmedizin Göttingen (DE), German Centre for Cardiovascular Research (DE), Kerckhoff Klinik (DE), Fraunhofer Institute for Digital Medicine (DE), Freie Universität Berlin (DE), Charité - Universitätsmedizin Berlin (DE), Ludwig-Maximilians-Universität München (DE)
Carl-Zeiss-Stiftung, Deutsches Zentrum für Herz-Kreislaufforschung, Bundesministerium für Bildung und Forschung, Klaus Tschira Stiftung
Openalex Percentile: Top 1%
Privacy-Preserving Technologies in Data
14.89
DOIOpen Access PDFOpenAlex
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.