A Framework for Machine-Actionable FAIR Assessments as a Tool to Measure FAIRification Success

Almost a decade after the FAIR principles have been worked out and published, their relevance for scientific data management has been recognized by most major stakeholders active in that area. However, while most research data repositories commit to FAIR, it can be questioned whether the level of machine-actionability of data that was envisioned in the original paper [1, section "The significance of machines in data-rich research environments"], that allows machines and software agents self-guided exploration and interpretation of data, has already been achieved. Concerns raised in a paper by Mons et al. [2] seem to be still at least partly valid. From these observations, the authors derive the conclusion that there is a need to develop a clearer and common understanding of what implementing the FAIR principles means at a technical level and to come to a more objective assessment of FAIRness of digital resources. Several approaches and tools for FAIR assessments of different kinds of resources have been published during the last few years [3], [4], [5], [6]. Most of them allow interactive evaluations on single, published digital resources, but they lack machine-actionable output. It is therefore difficult to do systematic mass analysis and to compare results of different tools. That however would be crucial to understanding different interpretations of the FAIR principles that are implicitly encoded in the tools and to develop common approaches towards implementation. In the FAIRagro project, we therefore built a FAIR assessment system that acts as a proxy to existing tools. It accepts data set identifiers and will hand them on to other FAIR assessment services to do the evaluation. Currently, it supports the tools from Wilkinson et al. [3] and f-uji [4], but its extensible architecture allows further systems to be connected. The output is unified into an RDF representation leveraging the data quality vocabulary (DQV) from W3C [7]. This provides an abstract framework to express different data quality metrics, dimensions and categories and thus allows for example to group determined FAIR metrics according to which principle they address. It can cover discrete scores as well as floating values. Using the tool, we did systematic mass assessments on the research data infrastructures represented in the FAIRagro project. Output data was stored in a Jena Fuseki triple store to do comparative analysis. Results for different data sets on a single repository were mostly homogeneous: for a single metric either all data sets succeeded or none of them. The output can be used to approach RDM system development in a more targeted manner addressing particularly the weak spots. It is also possible to identify FAIR principles where the metrics determined by different tools provide different outputs. These are indicators that for these, the research data community should work either on a better common understanding or provide explanations of why deviations might be acceptable. Future work planned includes providing concrete recommendations on what to improve in implementations to achieve better scores. We also plan on doing further analysis beyond the FAIRagro repositories based on the data available in the registry of research data repositories (re3data, [8]).