An RDM setup for Lattice QCD and other PUNCH4NFDI use cases

The primary datasets in Lattice Quantum Chromodynamics (QCD) are ensembles of gauge field configurations and reach volumes of order PB. The generation of these datasets by Markov Chain Monte-Carlo simulations usually requires significant high-performance computing resources and is often carried out by international research collaborations. Given the time and resources required, the datasets are reused across multiple research projects and often shared with the wider community. Thus, managing, storing, and providing standardized access to these datasets is a major concern. The International Lattice Data Grid (ILDG) was created in the early 2000s to address these challenges. It is organized as a federation of autonomous regional grids forming a single Virtual Organization (VO). ILDG enables distributed storage, consistent metadata management, and interoperability through standardized services and formats. Its mission strongly aligns with and predates the FAIR data principles: ensuring data is Findable, Accessible, Interoperable, and Reusable. The core infrastructure operated by each regional grid includes a metadata catalog (MDC), a file catalog (FC), and long-term storage elements (SEs). ILDG provides a common user management system and standardized specifications of the community-agreed metadata schemas (QCDml) and of the API of the core services to guarantee their interoperability across the regional grids. Since 2022 a modernization effort to upgrade ILDG's capabilities and adapt to evolving research needs, usually referred to as ILDG2.0 has been underway. A major technical challenge of this upgrade was the transition from grid certificate authentication to token-based access via Indigo IAM. This improves usability and also enables fine-grained access control for datasets that are under an (initial) embargo. Essential contributions towards ILDG 2.0 have been provided by the PUNCH4NFDI consortium, a German initiative funded under the National Research Data Infrastructure (NFDI) framework, with a focus on particle physics, hadron and nuclear physics, astronomy, and astro-particle physics. In particular, PUNCH4NFDI has led to the development of key components of a modular and distributed RDM system to fit also the needs of communities beyond lattice QCD. It comprises of: - a flexible metadata catalog that can be freely configured with multiple schemas and supports complex queries - a file catalog to handle administrative metadata for replicated and distributed data storage independently from the scientific (descriptive) metadata - additional interface services and user-friendly client tools, e.g. for harvesting, searching, and generation of the metadata. These modular components can then be flexibly integrated into a distributed RDM system optimized for specific use cases. In turn, the integration of these components into the ILDG has provided an important proof of concept for the practical implementation of fine-grained access control for data and metadata -- a feature that might be interesting in other PUNCH4NFDI developments and can also be integrated into the broader ecosystem of services being developed by the consortium.