Large language model generation of plausible semantic distractors for data science Parsons problems

Parsons problems have been widely recognised to support novice learners by reducing the cognitive demands of writing code from scratch through rearrangement of provided code blocks. A common variation includes distractors – erroneous blocks of code which do not belong to the correct solution. While Parsons problems have been widely studied for introductory programming, they remain underexplored in various domains including in data science education, where students may possess statistical knowledge but limited programming experience. Developing distractors is also time-consuming for educators. This study investigated the feasibility of using a large language model (LLM) to generate semantic distractors for data science Parsons problems. A set of distractors was generated for 315 Python-based data science Parsons problems and was evaluated for pedagogical plausibility, compilability, and output divergence. Validated distractors were then incorporated as exemplars in a refined prompt to generate a second set of distractors, which were evaluated using the same criteria. The findings demonstrate that LLMs can be used to develop semantic distractors for data science Parsons problems and refining the prompts using validated exemplars further improved the quality. This study contributes to the limited literature on data science Parsons problems, distractor design and LLM-supported development of these problems.

Authors

Institutions

Publication Details

Journal
ASCILITE Publications
Published
2026-11-17
DOI
https://doi.org/10.65106/apubs.2026.3251
Primary Topic
Teaching and Learning Programming
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

Large language model generation of plausible semantic distractors for data science Parsons problems

Leying Zou, Eduardo Araújo de Oliveira, Geela Venise Fabic Chee
3 links
ASCILITE Publications
Teaching and Learning Programming
article

Large language model generation of plausible semantic distractors for data science Parsons problems

Leying Zou, Eduardo Araújo de Oliveira, Geela Venise Fabic Chee
article en

Abstract

Parsons problems have been widely recognised to support novice learners by reducing the cognitive demands of writing code from scratch through rearrangement of provided code blocks. A common variation includes distractors – erroneous blocks of code which do not belong to the correct solution. While Parsons problems have been widely studied for introductory programming, they remain underexplored in various domains including in data science education, where students may possess statistical knowledge but limited programming experience. Developing distractors is also time-consuming for educators. This study investigated the feasibility of using a large language model (LLM) to generate semantic distractors for data science Parsons problems. A set of distractors was generated for 315 Python-based data science Parsons problems and was evaluated for pedagogical plausibility, compilability, and output divergence. Validated distractors were then incorporated as exemplars in a refined prompt to generate a second set of distractors, which were evaluated using the same criteria. The findings demonstrate that LLMs can be used to develop semantic distractors for data science Parsons problems and refining the prompts using validated exemplars further improved the quality. This study contributes to the limited literature on data science Parsons problems, distractor design and LLM-supported development of these problems.

ASCILITE Publications
The University of Melbourne (AU), Florey Institute of Neuroscience and Mental Health (AU)
Quality education
Teaching and Learning Programming
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon