Evaluating the Effectiveness of AI Integration in Software Engineering Education

Objectives The rapid spread of AI – especially large language models (LLMs) – is reshaping software engineering and raising questions for education. We examine the effectiveness of integrating AI into software engineering education. Participants This study involved 48 undergraduate students enrolled in a junior-level software engineering course. Study Methods We implemented an intervention, aligned with ACM/IEEE curriculum guidelines, that integrates an LLM-based prompt session with a conventional lecture. The lecture introduces the topic and explicit strategies for prompt design; the prompt session engages students in constructing, running, and iteratively refining prompts, analyzing LLM outputs, and reflecting on outcomes. A structured prompt-design framework is adopted to standardize context, inputs, tasks, and expected outputs to ensure clarity and consistent engagement. Findings A pilot in a junior-level course showed modest gains over lecture alone: on survey items repeated immediately after the lecture (before the prompt session) and again after the prompt session, scores improved by an average of 4.19%. Students rated the approach favorably, with scores ranging from 3.78 to 4.16 and an overall average of 4.02 out of 5, with the largest gains in prompt-construction skills, rising by 7.01%. By contrast, gains in identifying incorrect LLM outputs were only 3.67%, and improvements in understanding LLM limitations were just 2.22%. Conclusions Embedding LLM prompt sessions can supplement software engineering instruction but, in the study setting, produced limited benefits. The findings suggest that LLMs should be positioned as supportive tools, with emphasis on prompt design, critical evaluation of outputs, and expanded time for hands-on practice and assessment. Future work should include controlled, longer-term studies and alignment with industry needs to determine when and how LLM integration measurably improves learning outcomes.