Large Language Model-Powered Automated Assessment: a systematic review

The primary purpose of this systematic review is to examine peer-reviewed studies published between 2018 and 2024 on Large Language Model-Powered Automated Assessment (LLMPAA). The study investigates how LLMPAA systems are applied in educational contexts—particularly in essay and short-answer grading—and explores their effectiveness, challenges, and implications for human-AI alignment in assessment. Methodology: This systematic review follows the PRISMA 2020 guidelines. Studies were selected from five major academic databases: Web of Science, Scopus, IEEE Xplore, ACM Digital Library, and PubMed. The inclusion criteria focused on peer-reviewed articles that applied LLMs for automated scoring or grading in educational settings. A total of 49 studies meeting the criteria were analyzed in depth, covering dataset characteristics, model performance, evaluation metrics, and ethical considerations. Implications for Practice and Policy: The findings provide practical guidance for educators and policymakers regarding the implementation of LLMPAA systems. Insights from the review can inform decision-making related to digital assessment tools, especially in balancing scalability, reliability, and fairness. The study also highlights critical areas—such as bias, consistency, transparency, and data privacy—that must be addressed for responsible deployment of AI in education. Conclusion: This review contributes a comprehensive synthesis of current research on LLMPAAs, revealing both their potential to transform educational assessment and the limitations that must be resolved. It aims to support future innovations in automated assessment by offering evidence-based recommendations and identifying directions for further research.