Mastery Learning in CS1 with High Transparency Tests: Challenges for Fairness Among Task Variants

Context and motivation: There are several educational motivations for developing question banks with multiple variants of the same or similar task. One motivation might be that students can take tests at different times, or they are allowed several retakes of the same test. For instance, courses using self-paced mastery learning will typically have this feature, needing many variants of tests to avoid that students simply memorize answers. At the Norwegian University of Science and Technology, an introductory programming course for STEM teacher students is run according to a mastery learning paradigm. To mitigate test anxiety among students, a high level of test transparency is strived for with informal exercise tests that the students can use during test preparation, randomly pulling questions from the same question banks as used by the summative tests. After the 2023 run of the course the teacher found some tasks to have variants of unequal difficulty, which might be unfair in summative tests, and tried to fix this for the 2024 run of the course. Question / problem: This paper investigates whether there was a significant difference in difficulty between variants of the tasks, what were the causes for these differences, and whether the changes done between the first and second run of the course contributed to reducing the issues of unfairness. Principal ideas / results: This is done by analyzing the formative and summative test results for students who had consented to have their performance data used for research. Statistical analysis showed significant differences in difficulty for some variant groups, and the changes done for the second run of the course were indeed successful in reducing the variation in difficulty. Contribution: The paper analyses possible causes for the variation in difficulty, related to question framing, phrasing of answer options, and sequencing of code lines. The main contribution is to show that even minor changes to tasks can cause differences in performance, indicating that improvements to the task could reduce unfairness between variants. The paper concludes by identifying some factors to consider when designing tasks with many variants.