Statistical Modeling and Analysis of Electronic Examination Logs

This research investigates the effectiveness of randomized computer-based exams used in Carnegie Mellon's Introduction to Computer Systems course. Students received exams with 7-8 multiple choice and short answer questions that were automatically graded with regular expressions. Each question is randomly drawn from a pool of questions within its respective category. The exam system collects data about how each student progresses throughout their exam. This data [1] includes the order in which students view and answer the exam questions, the time spent viewing each question, and the score they have achieved at any given point during their exam. Our analysis of student scores quantifies and validates the fairness of these exams by reproducing prior methodology [2] that uses the MIRT multi-dimensional Generalized Partial Credit Model. The model allows us to estimate question difficulty, student ability, and question discrimination for any given student and exam question. Further analysis examines the correlation of higher-level student behaviors (e.g. order of solving questions) with ability and score. Key features include which question categories students spent the longest on, which ones students scored the highest on, and which ones the majority of students started the exam with. These insights were derived with statistical methods as well as supervised and unsupervised machine learning methods. This research supports the value of administering computer-based exams and informs future exam design. This research also motivates future interventions to test for identifiable behavioral features and students' test-taking strategies.