A Commentary on the Relationship between Pass/Fail Scoring and the Validity of Psychological and Educational Assessments: Using Medical Education as an Example

Background/Introduction: There have been numerous recent initiatives within medical education to convert multi-tiered academic assessments and the grades that summarize them to a dichotomous Pass/Fail (P/F) metric. Because the validity of an assessment depends upon how it is scored, an investigation of the psychometric outcomes of converting a multi-tiered assessment to P/F is needed. Methods: The psychometric and statistical validity implications of converting multi-tiered scores and grades to P/F are examined. How P/F scoring impacts test-based learning is also considered. Results: For the multi-tiered assessments used in medical education, there will generally be a large decrease in the variance of scores after converting to P/F. Statistical calculations demonstrate that compared to multi-tiered scores, P/F scores exhibit much smaller concurrent and predictive validity correlations and significantly attenuated reliability. Because P/F scores have a limited ability to correlate with other variables and provide very little information (i.e., low score reliability) regarding performance, the necessary pre-conditions for validity are practically absent after converting medical education assessments to P/F. Further, evidence suggests that using assessments with low validity will negatively impact learning. Discussion/Conclusions: There does not appear to be a sound psychometric or educational rationale for converting multi-tiered academic assessments to P/F.