Evaluating the Impact of Technical Flaws in Multiple-Choice Questions on Student Performance in Physiology

Graduate and professional level students are frequently evaluated through multiple-choice question (MCQ) examinations, a format that enables efficient assessment of diverse topics across large groups of students. While MCQs enhance the validity of exams, they may present challenges for test-takers. Technical item flaws in questions can introduce irrelevant difficulty or advantages for students with strong test-taking strategies, potentially compromising the fairness and effectiveness of the assessment. In this study, we investigated the influence of technical item flaws on test outcomes across two schools of medicine. We analyzed demographic factors and the relationship between the presence of item flaws and item performance. A total of 54 first-year medical students from the College of Medicine, University of Central Florida (UCF) and 32 first-year dental students and graduate students in the medical science program at Boston University Chobanian & Avedisian School of Medicine (BU) completed a 20 question physiology assessment and demographic survey. An evaluation instrument based on the National Board of Medical Examiners (NBME) item writing guide was employed to identify the item writing flaws in each question. Quantitative analysis included two independent sample t-tests for demographic factors, Spearman's rank correlation, and Mann-Whitney U test. The demographic factors studied were sex, student test-taking accommodations, race (White vs non-White), English proficiency (native vs non-native), birthplace (born in the U.S. vs not born in the U.S.), and home language (English vs non-English). Based on the evaluation instrument, we found 75% of the 20 question assessment contained at least one item flaw. No significant differences in test performance were observed between demographic groups at either institution. Additionally, there was no significant relationship between the number of flaws and item difficulty at either institution. However, dental and graduate level students exhibited a significant positive correlation between number of flaws and point biserial correlation, as well as a positive correlation between item difficulty and point biserial correlation. We were also able to identify that one of the flaws studied - the question lacks a focused, clear, or succinct stem or lead-in - can impact an item’s ability to discriminate between high and low performers. This study highlights the significant role that technical item flaws in MCQs can play in influencing test outcomes. The relationship between number of flaws and point biserial correlation suggests that flawed items may disproportionately favor high performing students, as these students are able to perform well despite the flaws. Additionally, items with unclear stems and lead-ins may provide unreliable indicators of student ability, as they can hinder the ability to differentiate between high and low-performing students. These results call attention to the importance of careful question design to ensure assessments are fair and accurately reflect student performance, particularly for diverse student populations. Note: Gloria Urrutia and Larissa Dixon contributed equally to this study. This abstract was presented at the American Physiology Summit 2025 and is only available in HTML format. There is no downloadable file or PDF version. The Physiology editorial board was not involved in the peer review process.