TrainTracks - federated learning for reproducible research on sensitive medical data

BACKGROUND: Reproducibility of computational algorithms is a challenging but crucial requirement for medical research and an important component of trustworthy training and application of AI algorithms. Federated Learning (FL) is commonly used to enable privacy-preserving AI in medical research. One prerequisite of reproducibility is traceability. A majority of publications on traceable FL platforms leverage blockchain technology to achieve traceable FL. In healthcare settings, resource-efficient alternatives to blockchains are possible; however, their traceability features require separate design considerations. METHODS: To meet the growing demand for reproducible AI, as outlined in guidelines published by governing bodies such as the European Commission, we propose a novel concept TrainTracks. TrainTracks extends the established Personal Health Train (PHT) Platform for Analytics and Distributed Machine Learning for Enterprises (PADME) to support reproducible and traceable federated learning in medical research. PADME already partially supports tracing the FL and changes to the analysis algorithms used in the FL projects. We extend PADME by adding privacy-preserving change tracing of the data, metadata, and computational experiment execution through integrating it with the tools specialized in distributed data management (DataLad and MetaLad). Finally, we evaluate the proposed concept against the detailed list of requirements to analyze the advantages of the TrainTracks and the need for further design improvements. RESULTS: Evaluation of the TrainTracks concepts' compliance with a checklist for reproducible AI showed that TrainTracks improves the original PADME platform in 15 points out of 47 points applicable to FL. The greatest improvement was in data reproducibility, improving 10 of 12 points from no support to full support for automatic information extraction. No improvements were made to method reproducibility, aside from introducing a dedicated reproducibility repository. Experiment reproducibility saw upgrades in 5 of 30 applicable points, mainly through workflow and code traceability. CONCLUSIONS: Our concept of combining FL technology with a data versioning tool provides a structured, automated workflow that traces the FL algorithm itself, delivered algorithms, and the used data. TrainTracks demonstrated high compliance with recommendations for reproducible AI experiments, methods, and data. Our work emphasizes the importance of complete FL process traceability, as all considered aspects individually contribute to the reproducibility of the medical research. Tracking dataset versions in FL is crucial for dynamic application areas, such as medical research, where new data, e.g. electronic health records, are continuously recorded and added.