Good educational practices with digital technologies: validation of an instrument for direct classroom observations

Abstract In this study we present a process for designing and validating an observation guideline based on the Activity Theory (AT) for analyzing educational practices that incorporate the use of digital technologies (DT). The main objective was to develop a data-collection instrument for identifying and analyzing good educational practices that demonstrate the pedagogical use of DT through direct observation undertaken from basic education to higher education. To do so we adopted a Design-Based Research (DBR) approach structured around iterative cycles of analysis, design, development and evaluation. This observation guideline was based on a systematic literature review in which 26 variables organized around the components of AT were identified. We then conducted a validation process using the Delphi method in three rounds (R1 = 23, R2 = 20, R3 = 9) involving a panel of experts in the fields of education and technology. Using this technique, we were able to reach a consensus on observable indicators, assess their clarity, relevance and appropriateness, and revise those that required improvement. The final version of the guideline consisted of 14 indicators that met the predefined consensus threshold (CVI ≥ 78%). Finally, the instrument was piloted in real educational settings within eight contexts distributed across four educational stages—early childhood, primary, secondary, and university—each represented by two public institutions which enabled adjustments to be made in content, design and usability. Inter-rater reliability analyses using Cohen’s weighted Kappa and the Intraclass Correlation Coefficient (ICC) showed substantial to excellent agreement between observers, confirming the robustness of the guideline. Our results show a high degree of validity and feasibility in the implementation of the observation guideline, which confirms its effectiveness as a tool for analyzing teaching and learning dynamics that incorporate the use of DT across diverse educational levels.