Automating New Zealand High School Programming Assessment - Teachers' Experiences

In Aotearoa New Zealand, an updated version of the introductory programming standard has been released.In an attempt to improve the equity of programming education, we have developed an automated assessment system, known as NCEA-CodeRunner, designed to assist with assessing the criteria of the standard.This tool was released in early 2024, with 93 high school teachers signing up to express interest in the system, of which 36 used the system to assess the updated standard.This paper reports on a survey of both teachers who opted to use the system, and teachers who did not.We find that there was a wide array of reasons for teachers to not use the system, with the most common being concerns regarding complexity, and having pre-existing assessment plans.However, many of these teachers are considering using the system in the future.We then discover that the automated assessment system, intended to make it easy for novice programmers to assess the standard without becoming an expert, appealed primarily to experienced teachers, including those with experience programming and assessing the old version of the standard.We also find that teachers who used the system generally viewed it favourably.They reported that it saved them time, was not demanding to use, made it easier to be fair and consistent in marking, and were willing to use it again in future years.We also report on issues with the system, specifically around conforming to external moderation performed by a governing body.