Is Self-Mark dependable in Very Short Answer Question formats among pre-clinical medical students?

Introduction: Very Short Answer Questions (VSAQs) minimise cueing and simulate actual clinical practice more accurately than Single Best Answer Questions, as multiple-choice options might not be realistic. Phramongkutklao College of Medicine has developed a Self-Marked VSAQ (SM-VSAQ) for formative assessments. This study determines the validity and reliability of the SM-VSAQs. Methods: Ninety-four third-year pre-clinical students took two occasions of 10-question SM-VSAQ exams regarding cardiovascular drugs. Each question consisted of two steps: (1) clinical vignettes with questions and (2) expected answers with scores, self-marking, and feedback comprehension. Scores ranged from 0.00 to 1.00 in 0.25 increments, though not every increment was applied to all questions. A distribution of the rating agreement between students' and teacher’s ratings was presented to determine criterion-related validity and inter-rater reliability. Results: Criterion-related validity revealed 90.64% and 93.19% of the ratings demonstrated exact agreement between students' and teachers' ratings, with an inter-rater reliability of 0.972 and 0.977 for the first and second occasions, respectively (p=0.001). The exact agreement was relatively lower on the first occasion for questions with more diverse expected answers (85.11%, r=0.867, p=0.001) and drugs requiring their specific full names for a perfect mark (74.47%, r=0.849, p=0.001). While questions with specific guides do not require complex answers, they received a higher exact agreement. Conclusion: The SM-VSAQ format effectively combines guided answers with the VSAQ model. The agreement with teacher-rated is excellent. Marking discrepancies rooted in misconceptions underscores the importance of teacher feedback in improving self-grading in formative assessments. Regular self-assessment practice is recommended to enhance grading accuracy.