From Black-Box Automated Scoring to Engaging Feedback: A Docimological-Pedagogical Pipeline for Assessment with Local LLMs

The use of artificial intelligence in assessment raises questions about how automated judgements relate to learning goals and how students can use the feedback they receive. This paper presents the first stage of a broader research project on the assessment of written responses in Italian provided by local large language models (LLMs). We conducted a purposive narrative review of ten technical publications, retrieved through ACL Anthology, arXiv and publishers’ websites before1 October 2026, utilizing literature on feedback and assessment practices. The synthesis shows that scoring performance depends on the task, the human reference and the evaluation setting, while evidence concerning feedback use remains limited. These findings inform a proposed five-layer pipeline in which explicit criteria and independently rated responses provide the basis for semantic analysis, local-model evaluations and teacher review. The pedagogical rationale is to support judgements whose grounds can be verified and feedback that learners can use in their revision. As the pipeline has not yet been tested, we outline a validation program addressing scoring agreement, the quality of reviewed decisions, teacher workload and students’ interpretation and use of feedback.

Authors

Publication Details

Journal
Aisberg (University of Bergamo)
Published
2026-11-20
Primary Topic
Student Assessment and Feedback
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

From Black-Box Automated Scoring to Engaging Feedback: A Docimological-Pedagogical Pipeline for Assessment with Local LLMs

Marco Giganti, Marco Lazzari
Aisberg (University of Bergamo)
Student Assessment and Feedback
article

From Black-Box Automated Scoring to Engaging Feedback: A Docimological-Pedagogical Pipeline for Assessment with Local LLMs

Marco Giganti, Marco Lazzari
article en

Abstract

The use of artificial intelligence in assessment raises questions about how automated judgements relate to learning goals and how students can use the feedback they receive. This paper presents the first stage of a broader research project on the assessment of written responses in Italian provided by local large language models (LLMs). We conducted a purposive narrative review of ten technical publications, retrieved through ACL Anthology, arXiv and publishers’ websites before1 October 2026, utilizing literature on feedback and assessment practices. The synthesis shows that scoring performance depends on the task, the human reference and the evaluation setting, while evidence concerning feedback use remains limited. These findings inform a proposed five-layer pipeline in which explicit criteria and independently rated responses provide the basis for semantic analysis, local-model evaluations and teacher review. The pedagogical rationale is to support judgements whose grounds can be verified and feedback that learners can use in their revision. As the pipeline has not yet been tested, we outline a validation program addressing scoring agreement, the quality of reviewed decisions, teacher workload and students’ interpretation and use of feedback.

Aisberg (University of Bergamo)
Student Assessment and Feedback
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon