Development and validation of a competency-based evaluation framework for clinical medical trainees in the era of artificial intelligence: a mixed-methods study in China

The integration of artificial intelligence (AI) into clinical practice is reshaping the competency requirements for medical trainees. Yet, validated evaluation instruments aligned with outcome-based education (OBE) frameworks remain scarce. We conducted a sequential mixed methods study to develop and preliminarily evaluate an OBE-based competency assessment matrix for clinical medical trainees in China. The framework was derived from national and international competency standards and refined through a three-round Delphi process with 16 medical education experts. Empirical evaluation involved 276 respondents including residents, postgraduate students, and clinical educators who completed the finalized 72-item instrument via a digital assessment platform. Reliability and exploratory structural characteristics were examined using Cronbach’s α, exploratory factor analysis (EFA), and inter-item correlation matrices. Subgroup differences were examined descriptively and visualized with radar plots. The Delphi panel reached consensus on 72 items across three domains—Importance, Feasibility, and Clarity—with progressive convergence (Kendall’s W ranging from 0.65 in Round 1 to 0.74 in Round 3). The resulting scale showed excellent internal consistency (Cronbach’s α = 0.928) and strong sampling adequacy (KMO = 0.884). Bartlett’s test of sphericity was highly significant (χ2 = 421.35, df = 28, p < 0.001), confirming the suitability of the data for structural exploration. EFA of aggregated domain scores yielded a three-component pattern that cumulatively explained 74.5% of the variance. The resulting loading profile suggested meaningful contributions of Importance, Feasibility, and Clarity, offering exploratory support for the proposed domain-level structure. Radar plots revealed systematic but role-dependent differences: faculty emphasized Importance, residents prioritized Feasibility, and postgraduates rated Clarity slightly higher. This study provides a context-sensitive evaluation matrix with encouraging initial psychometric evidence, tailored to the evolving demands of AI-informed clinical education. The framework offers a promising platform for competency assessment and curriculum development in Chinese teaching hospitals and may serve as a reference model for other AI-integrating medical education systems, while highlighting the need for confirmatory factor analysis in independent samples to more definitively establish its dimensional structure.