When technical is easy: Using capability-level rubric data to surface the gap between production and judgement

Generative AI has not changed the need for assessment, but it has changed what certain forms of assessment can reliably tell us about student performance. When students can access AI support for idea generation and technical implementation, what assessment should measure becomes an open question (Perkins et al., 2024) and aggregate course grades may provide little insight to distinguish strong conceptual work from competent production. We report on a preliminary analysis of rubric data from a postgraduate data visualisation course (n =175) where AI use for idea generation and coding support was explicitly permitted and treated as a normal part of the workflow. The course assessment was redesigned around capability-level rubrics to capture what matters most when technical production can be AI-assisted. The assessment design was informed by evaluative judgement theory (Bearman et al., 2024; Tai et al., 2018), where the capacity to recognise and justify quality is positioned as a graduate capability developed through repeated engagement with standards. The course used descriptive capability rubrics with five levels from Foundation to Exceptional, applied across multiple assessments and a final project portfolio. As Gyamfi et al. (2022) demonstrate, descriptive levels preserve the interpretive work that evaluative judgement requires rather than orienting students toward compliance. A separate competency review assessed critique, interpretation, and application of visualisation principles under exam conditions without AI access. The final grade distribution looked similar to previous iterations of the course, but the capability-level rubric data had a more interesting story to tell. Distinct student profiles emerged, including those whose technical implementation capability exceeded their ability to communicate a coherent data story, suggesting a gap between procedural and deeper conceptual capability (Biggs & Tang, 2011). A similar pattern was visible in a dedicated storytelling assessment, where strong visuals were not consistently matched by strong narrative performance. These within-student differences were largely invisible in aggregate grades. The competency review provided an independent measure of evaluative judgement: what students could do without AI assistance or visualisation making tools. Competency review performance showed the strongest though modest association with the portfolio data story criterion, suggesting that the ability to interpret and communicate was the more reliable indicator of independent judgement (Bearman et al., 2024). We argue that rubric data, already generated through normal marking processes in most LMS environments, constitutes an under-utilised source of learning analytics. Examined at the level of capability profiles rather than aggregate scores, this approach can surface distinctions between production and judgement that grades conceal across many discipline contexts, providing actionable insights that connect assessment evidence to teaching decisions (Kennedy et al., 2025). Rather than framing AI as a threat to assessment integrity, we ask a different question: if technical production is increasingly supported, which capabilities remain the most trustworthy evidence of what students know? As argued by Corbin et al. (2026), deciding which forms of evidence deserve the greatest trust may matter more than detecting whether AI was used. We show that capability-based rubric data may be one such form of trustworthy evidence.

Authors

Institutions

Publication Details

Journal
ASCILITE Publications
Published
2026-11-17
DOI
https://doi.org/10.65106/apubs.2026.3427
Primary Topic
Student Assessment and Feedback
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

When technical is easy: Using capability-level rubric data to surface the gap between production and judgement

Nicholas Fewster–Young, Malgorzata Korolkiewicz
3 links
ASCILITE Publications
Student Assessment and Feedback
article

When technical is easy: Using capability-level rubric data to surface the gap between production and judgement

Nicholas Fewster–Young, Malgorzata Korolkiewicz
article en

Abstract

Generative AI has not changed the need for assessment, but it has changed what certain forms of assessment can reliably tell us about student performance. When students can access AI support for idea generation and technical implementation, what assessment should measure becomes an open question (Perkins et al., 2024) and aggregate course grades may provide little insight to distinguish strong conceptual work from competent production. We report on a preliminary analysis of rubric data from a postgraduate data visualisation course (n =175) where AI use for idea generation and coding support was explicitly permitted and treated as a normal part of the workflow. The course assessment was redesigned around capability-level rubrics to capture what matters most when technical production can be AI-assisted. The assessment design was informed by evaluative judgement theory (Bearman et al., 2024; Tai et al., 2018), where the capacity to recognise and justify quality is positioned as a graduate capability developed through repeated engagement with standards. The course used descriptive capability rubrics with five levels from Foundation to Exceptional, applied across multiple assessments and a final project portfolio. As Gyamfi et al. (2022) demonstrate, descriptive levels preserve the interpretive work that evaluative judgement requires rather than orienting students toward compliance. A separate competency review assessed critique, interpretation, and application of visualisation principles under exam conditions without AI access. The final grade distribution looked similar to previous iterations of the course, but the capability-level rubric data had a more interesting story to tell. Distinct student profiles emerged, including those whose technical implementation capability exceeded their ability to communicate a coherent data story, suggesting a gap between procedural and deeper conceptual capability (Biggs & Tang, 2011). A similar pattern was visible in a dedicated storytelling assessment, where strong visuals were not consistently matched by strong narrative performance. These within-student differences were largely invisible in aggregate grades. The competency review provided an independent measure of evaluative judgement: what students could do without AI assistance or visualisation making tools. Competency review performance showed the strongest though modest association with the portfolio data story criterion, suggesting that the ability to interpret and communicate was the more reliable indicator of independent judgement (Bearman et al., 2024). We argue that rubric data, already generated through normal marking processes in most LMS environments, constitutes an under-utilised source of learning analytics. Examined at the level of capability profiles rather than aggregate scores, this approach can surface distinctions between production and judgement that grades conceal across many discipline contexts, providing actionable insights that connect assessment evidence to teaching decisions (Kennedy et al., 2025). Rather than framing AI as a threat to assessment integrity, we ask a different question: if technical production is increasingly supported, which capabilities remain the most trustworthy evidence of what students know? As argued by Corbin et al. (2026), deciding which forms of evidence deserve the greatest trust may matter more than detecting whether AI was used. We show that capability-based rubric data may be one such form of trustworthy evidence.

ASCILITE Publications
Adelaide University (AU), The University of Adelaide (AU)
Quality education
Student Assessment and Feedback
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon