Certainty-based marking in multiple-choice assessments in physiology: a web-based implementation using an AI assistant

Certainty-based marking (CBM) requires students to indicate their certainty levels alongside their answers. CBM has been shown to enhance self-assessment and metacognitive awareness. This study aimed to explore the implementation of CBM in multiple-choice assessments in physiology. The CBM assessment tool was developed with an artificial intelligence (AI) assistant, Claude 3.5, with prompts focused on functional rather than technical requirements. The assessment consisted of 15 multiple-choice questions (MCQs), which were administered as a pretest and posttest during a small group teaching session to first-year medical students. Following the assessment, students completed a survey to evaluate their perceptions regarding the format, knowledge-gap identification, and overall acceptability. Answers from 195 students were analyzed, and significant improvements were observed in performance measures and certainty indices from the pretest to the posttest. Most students (80.9%) found the certainty scale beneficial, and 78.3% changed their answers after reflecting on their certainty. CBM demonstrated metacognitive benefits, with 86.4% of students better recognizing their knowledge gaps and 85.8% feeling more aware of their learning progress. About 73% of students preferred the CBM format and expressed greater engagement (82.8%) than traditional MCQs. CBM implemented through a web-based platform functioned as an assessment tool and an instructional intervention that enhanced students' metacognitive awareness and self-monitoring skills in physiology education. Our study focused on a single physiology topic and showed improvements in knowledge retention and certainty calibration. However, further longitudinal studies across multiple topics are needed to determine whether students maintain these self-assessment skills over time.<b>NEW & NOTEWORTHY</b> To introduce certainty-based marking (CBM) to novice students, a custom web-based multiple-choice question (MCQ) test was developed with assistance from an artificial intelligence (AI) tool. This enhanced accessibility and allowed for data collection to evaluate and analyze student performance. The integration of AI in creating this assessment tool highlights the potential of technology to improve educational practices, especially in designing various assessment strategies.