Exploring zero-shot essay scoring: from feature-based to LLM-based approaches

Automated Essay Scoring (AES) plays a crucial role in educational assessment, yet building reliable systems often requires large amounts of prompt- and trait-specific labeled data. As a result, many studies rely heavily on a single public dataset, ASAP++, which limits their generalizability. In this paper, we explore diverse zero-shot AES approaches that are both prompt-agnostic and multi-trait, thereby reducing reliance on labeled data. Our study examines large language models (LLMs), BERT-like fine-tuned models, and simple feature-based methods. Furthermore, we propose two novel zero-shot frameworks: CAnUSe (Comparative Assessment and subsequent Uncertainty-aware Self-training) and EASY (Essay Assessment with Simple Yardsticks). CAnUSe combines LLM-based pairwise comparisons with uncertainty-aware self-training using a BERT-like single-essay scoring model, while EASY employs only three intuitive features-essay length, vocabulary diversity, and grammar quality. Evaluation on three benchmark datasets-ASAP++, Ellipse, and TOEFL11-shows that our frameworks achieve state-of-the-art performance, demonstrating strong generalizability across datasets. Remarkably, despite operating in a zero-shot setting, our models approach the performance of fully supervised models trained with approximately 10K labeled samples. Further analysis provides practical insights for building real-world AES systems.