Data Science Masters Thesis :Building Bridges in Research: Integrating Provenance and Data Management in Virtual Research Environments.
Machine learning workflows in modern scientific research are often fragmented acrossmultiple tools and environments, resulting in scattered or incomplete metadata.This fragmentation makes it impracticable to determine how models were trained,which datasets were utilized, and what decisions impacted experimental results.As a result, transparency, reproducibility, and auditability deteriorate, limiting thelong-term reliability and reusable nature of machine learning research.This thesis examines into how semantic provenance techniques might be includedinto Virtual Research Environments (VREs) in order to address these problems. Bydefining key metadata needs across the machine learning lifecycle, we create a formalmapping to community standards like as FAIR, FAIR4ML, PROV-O, Croissant, andMLSea. Based on this basis, we suggest a strategy for integrating diverse metadatasources into a semantically rich representation.This approach is implemented within a modular framework that combines Jupyter-Hub, GitHub, DBRepo, and Invenio, with MLflow support for semi-automatedmetadata gathering. Key aspects including dataset provenance, training setups, run-time environment, and justification information are exported in machine-readableforms (JSON-LD and RDF/XML). A Streamlit-based dashboard makes it easier tovisualize, compare experiment runs, and do SPARQL-based provenance queries.
Publication Details
- Journal
- Zenodo (CERN European Organization for Nuclear Research)
- Published
- 2025-01-01
- DOI
- https://doi.org/10.5281/zenodo.16890558
- Primary Topic
- Theoretical and Computational Physics
- Type
- article
- Field-Weighted Citation Impact
- 0.00