Failures in Reliably Assessing Program Code Readability

Writing correct code is an important part of learning to program, but other attributes such as readability also matter for good code, and thus it is desirable to assess them while teaching programming. In this paper we contrast two ways of grading readability: traditional grading (taken from datasets in three different previous studies), and the use of Comparative Judgement (in a new study with 20 graders on 80 projects). We find that both grading approaches are very unreliable, calling into question whether readability of program code can feasibly be graded at all.