Leveraging Large Language Models to Automatically Investigate Core Tasks Within Undergraduate Engineering Work-Integrated Learning Experiences

This full research paper aims to investigate methods for systematically identifying core tasks within undergraduate engineering work-integrated learning (WIL) opportunities, such as internships and co-ops. It achieves this by automatically analyzing WIL opportunities using transformer models. A dataset of 4,833 engineering internship postings from the last ten years was obtained through a partnership with the University's Career Connections Center. From this, a subset of 374 aerospace engineering internships, yielding 1,913 unique job tasks, was extracted for human labeling. We applied the Llama 2 architecture, a sophisticated pre-trained LLM, to extract a list of specific responsibilities and tasks from the internship postings. The job tasks were used to train an automated classification system to map each task to the established seven ABET student outcomes. Each job task was human-labeled by three subject matter experts, achieving a high level of inter-rater reliability of 0.998, according to Krippendorff's alpha. RoBERTa resulted in the optimal model indicating a label ranking average precision of 0.892 on the validation set and 0.857 on the testing set. Our findings provide novel insights into understanding the evolving skill expectations of undergraduate interns, offering a basis for tailoring engineering education to address these demands. Furthermore, the automated analysis of internship tasks demonstrates the potential for a scalable way to address the gap in understanding the core responsibilities within WIL experiences.