Human-Centred AI for Data Management Plan Assessment and Support

Research libraries are increasingly expected to safeguard responsible research data practices under conditions of technological uncertainty, rising administrative burden, and constrained resources. In this context, a central question is whether generative artificial intelligence can reliably support expert-led research data governance without undermining quality, accountability, or trust. This presentation addresses that question by empirically examining how AI-generated assessment of Data Management Plans (DMPs) can support rather than replace professional judgement in research data management. We present a study of DMP Analyser, a prototype AI tool developed by the authors to support expert review of DMPs using established evaluation criteria. The system applies a modular analysis pipeline to produce structured and traceable outputs and explicit audit trail from each judgement to sentence-level evidence in the original DMP. This design enables experts to inspect, validate, and contest AI outputs, positioning the tool as a decision-support system rather than an automated evaluator. The analysis pipeline is configurable and can utilise different large language models and evaluation rubrics, enabling institutions to apply shared criteria consistently and to accumulate an institutional knowledge base grounded in expert-validated DMP assessments. The study analyses a corpus of DMPs submitted to the 2025 Research Council of Finland funding call from Tampere University and the University of Turku. All funded projects are required to submit a DMP, which is reviewed by institutional data support professionals who provide written feedback based on a shared evaluation rubric. This setting offers a realistic governance context in which expert judgement is formally embedded and documented. Following expert assessment, the same DMPs are independently analysed using the DMP Analyser between January and May 2026, which produces structured evaluations aligned with the criteria used by human reviewers. AI-generated assessments were then systematically reviewed by the research team and compared against expert feedback and the original DMP texts. To support this analysis, the DMP Analyser was extended to store expert comments, AI outputs, and validation decisions, enabling the calculation of quantitative indicators of agreement and divergence across individual assessment criteria. This allows us to examine how the reliability and usefulness of AI support vary by topic and type of judgement, and to identify where human oversight remains essential. The findings highlight both the potential and the limitations of AI-assisted DMP support. While AI performs well on structurally explicit criteria, it struggles with context-sensitive judgements and ambiguous descriptions. Rather than treating this as a benchmark of AI performance, the study demonstrates a human-centred approach in which such limitations are made visible, traceable, and actionable for expert reviewers. The contribution lies in showing how generative AI can function as a transparent and verifiable decision-support infrastructure that strengthens professional judgement and institutional responsibility, rather than replacing them.