remarks - Machinery for Marking Student Work

remarks is an open-source suite of tools for marking student work. It has been in moderate use for assessing semi-structured submissions in a handful of Computer Science courses with hundreds of students since 2016. The contemporary approach to systematically collaborate on such assessments is to use general-purpose administrative software (e.g., spreadsheets, text files or documents, Emacs Org-mode). Since this software has not been expressly designed for the purpose, it tends to require non-trivial technical skill to achieve but mediocre technical support for the assessment process. In particular, it is hard to achieve (1) support for criteria-based analytic marking, without precluding (2) holistic, free-form justifications for the given marks, while enabling (3) decentralized collaboration with a reliable and transparent synchronization mechanism. remarks has been expressly designed to meet these criteria. Guided by the theory of assessment and evaluation in higher education, it has grown from a series of scripts surrounding general-purpose administrative software to a dedicated tool suite for marking student work. With remarks, assessing a unit of work constitutes filling in a document whose structure is guided by, but not limited to a chosen marking scheme. Assessors can use a contemporary source code revision control system (e.g., Git) to collaborate on these documents; making conflict resolution explicit and unsurprising. Using one document per unit of work, further reduces conflicting edits, in general. The remarks tool can then derive summative and descriptive assessments from such documents. We describe and justify the design of remarks, report on our experiences with the approach, and invite you to help develop the ideas further.