Needs for clinical reasoning assessment methods and automatic question generation: An international survey

Aim: Written assessment methods are widely used to assess clinical reasoning in undergraduate health professions education, yet little is known about the specific approaches used and educators’ needs. Although automatic question generation, especially AI-based, offers promising efficiency gains, the extent of its application in clinical reasoning assessment remains unclear. The aim of this study is to determine the needs for written clinical reasoning assessment methods and automatic question generation in undergraduate health professions education. Methods: An international web-based survey was conducted between February and June 2024. Participants were health professions educators knowledgeable about written assessment methods in their programs. The survey covered current practices, perceived importance, barriers, and needs related to written clinical reasoning assessment methods and automatic question generation. Parametric and non-parametric statistical tests were used for the analysis. Results: 51 programs from 33 countries participated. Multiple-choice questions (MCQs) were the most widely used method (100% of programs). Short-answer questions were perceived as most important for both formative and summative assessment. Main barriers in using the written assessment methods were lack of know-how (81.6%) and insufficient time (79.6%). “Differential diagnosis” and “management and treatment” were the most commonly assessed clinical reasoning components, while ethical aspects, patient perspective, and interprofessional collaboration were the least assessed. Only 58.1% of programs provide feedback on MCQ results to students, with varying levels of detail. Awareness and use of automatic question generation methods were low, both for template-based and non-template-based (artificial intelligence) methods. Educators showed willingness to participate in training on assessment methods and automatic question generation. Conclusion: While MCQs are ubiquitous, there is recognition of the importance of other methods such as short-answer questions. Significant barriers exist in benefiting from various assessment methods. There is a clear need for faculty development and greater awareness of innovative solutions, such as automatic question generation to enhance efficiency of assessment practices in medical schools.