D5.1 Pilot monitoring and evaluation methodology

This “Pilot monitoring and evaluation methodology” report is the first deliverable D5.1 of HECOF Work Package 5: “Pilot testing, evaluation and impact assessment”. Objective of this document is the initial planning of the HECOF piloting, evaluation, and monitoring in strong collaboration with WP4the agile development to initiate HECOF phase 3 “Monitoring and evaluation”.HECOF piloting will be implemented at two pilot universities in the field of chemical engineering.• Pilot 1: NTUA, chemical extraction process• Pilot 2: POLIMI, bioreactorHECOF follows a state-of-the-art hybrid development methodology [2] with Lean/UX in WP2, (see D2.1), learning design in WP3 (see D3.1 and D3.2), and “Agile development of HECOF system” in WP4. HECOF will be evaluated by two evaluations. Objective of evaluation 1 is to gather user feedback for the agile development and collect performance data from the HECOF system. Objective of evaluation 2 is to assess the outcome and impact of HECOF.The HECOF evaluation and monitoring methodology employs a mixed-methods approach, combining quantitative metrics from system performance and learning analytics with qualitative feedback from users. The methodology integrates current research for AI-adaptive learning and VR training to assess the system’s impact [3], [4], [5]. Six evaluation dimensions are discussed: a) Technical functionality, b) System performance: system accuracy performance, c) Pedagogical impact, d) Satisfaction and perceived usefulness, e) Usability Evaluation, f) Social Impact and Presence. Implementation of HECOF phase 3 “Monitoring and evaluation” is organized in eight subphases:● PH3.0: Planning of methodology (WP5, T5.1 NURO) D5.1 M21● PH3.1: Preparation of pilot activities (WP5, T5.2, E5.1 & E5.2 SIMAVI)● PH3.2: Users Training (WP5, T5.3, E5.2 & E5.4 ADAPTEMY)● PH3.3: Evaluation 1 running and monitoring (WP5, T5.4 SIMAVI)● PH3.4: Analytics of the outcome and refinement of D2.1 & D5.1 by D5.2 M24 (WP5, T5.5 NURO)● PH3.5: Agile development of MVP2● PH3.6: Evaluation 2 running and monitoring (WP5, T5.4 SIMAVI)● PH3.7: Analytics of the outcome and impact assessment by D5.3 M30 (WP5, T5.5 NURO. WP5 will guide through these phases, based on input from WP2, WP3, and in collaboration with WP4. T5.2 Preparation will refine this initial planning and deploy the first version of the HECOF system in PH3.1. Combined events with T5.3 User Training will introduce the pilot partners with the HECOF platform and set up the first evaluation in PH3.2. In this phase the VR exercises will be refined by the educators in collaboration with the developers. PH3.3 will conduct the implementation of the first evaluation. In PH3.4 outcome of the first evaluation will be reported by D5.2 “First evaluation of pilot activities & user requirements refinement” as refinement of D2.1 and this report. Based on this PH3.5 will finalize the HECOF System. The final version of HECOF will be assessed by the second evaluation in PH3.6. Findings of WP5 will about user satisfaction and system performance of the HECOF system will be reported with D5.3 “Final evaluation and impact assessment". This report will provide insights to technical aspects, pedagogical impact, satisfaction, perceived usefulness, usability, and social impact for the HECOF approach of a novel AI driven adaptive and immersive learning environment.This will achieve milestone MS7 “Second round of pilot activities are implemented”.