Strictly Proper Scoring Rules in an Absolute Grading Environment - Preliminary Findings

Strictly proper scoring rules are used to elicit a person's true probability beliefs about an uncertain outcome.The application of strictly proper scoring rules to grading in an academic environment is not new and is typically restricted to classes centered on Decision Analysis.For the purpose of explanation, a typical application of strictly proper scoring rules in academic grading would be as follows: assume that a multiple choice question with four possible answers has correct answer "D" and is worth one point.The traditional technique requires students to select one right answer, so if a student answers "D", the student receives a 1 or a 0 for all other answers.Conversely, a strictly proper scoring rule requires the student assign probabilities that each possible answer is correct, say A=0.1, B=0.2, C=0.05, D=0.65.The student's score depends on the scoring rule applied.Under the logarithmic scoring rule, the student would receive ln(0.65) points or -0.43.The scores are obviously bounded by (-, 0].Usually, the instructor rank orders students' scores and then assigns final grades.This situation can be extremely punitive for students who assign a low probability to a correct answer, and only slightly rewarding for those who submit their true understanding of the problem.Alternatively, the quadratic scoring rule allows a range of scores for the "correct" answer but is bounded between -1 and 1 allowing the instructor to similarly rank the scores.We discuss a modification of the quadratic rule applied at the United States Military Academy in our Decision Analysis course.In our approach, we are restricted to an absolute grading requirement -the grade a student earns is not curved in any way.We explore the trade off between information gained about the students' true beliefs and points awarded.We examine initial student feedback and compare probabilistic grades to the hypothetical traditional multiple choice grades.Finally, we explore options for integrating strictly proper scoring rules into other engineering courses.