A Federated Approach to Overcome Data Integration Challenges Across Research Domains

In the era of big data we are seeing a growing awareness of the importance of FAIR data [1], but nevertheless the integration of heterogeneous datasets across diverse research domains remains a significant challenge, particularly in the context of decentralized and autonomous research data infrastructures (RDIs) [2]. The design and implementation of concepts to overcome this challenge with a particular emphasis on improving interoperability of multiple domain data is an important objective of many NFDI consortia. One approach was initially developed for homogenizing the various RDIs with the agrosystems science community of the FAIRagro consortium [3], [4]. This was done in close collaboration with other NFDI consortia and international initiatives facing similar challenges to ensure that current developments are considered and broad integration with other data ecosystems is ensured. The core component of this concept is the so-called middleware, which acts as a federated service, enabling seamless and harmonized data integration across disparate infrastructures while preserving their operational and organizational autonomy. This is achieved through a combination of standardized metadata exchange, harmonized data access protocols, and the use of FAIR Digital Objects (FDOs) [5], [6] to ensure data integration and interoperability. The iterative development process started with an initial harvesting architecture for metadata dumps formatted as JSON files and based on embedded Schema.org metadata. This provided a basic homogenization for a rapid onboarding of diverse RDIs and allowed the start of the parallel development of different consumer services like a comprehensive search hub and a workflow sharing & execution platform. Subsequently the concept was extended by incorporating the increasing acceptance and usage of FDOs within different NFDIs. Leveraging the Annotated Research Context (ARC) framework [7], which based on the agnostic ISA model [8] for comprehensive, semantic metadata descriptions and the GitLab platform for the data handling, ensures that datasets coming from different RDIs can be integrated on a high level. Additionally, different kinds of data access procedures from the diverse RDIs, including public, protected or on-request paths as well as technical developments within the NFDIs given by the IAM4NFDI base service [9] were considered to design an authentication and authorization concept to be integrated in the overall architecture. This allows to intersect data from different domains and make them ready for downstream processing, which facilitates unlocking new data spaces and gaining new insight. By focusing on a federated, scalable and standards-based architecture the middleware approach offers a promising solution for improving data interoperability across domains [10]. It can serve as a blueprint for other consortia as well as provides the potential to create several datahubs in a bottom-up way, which can in future be easily connected offering the different community the access to a broader data ecosystem, which unites datasets from diverse domains.