Assessment of engineering graduate attributes: A case study on modeling and analyzing heterogeneous data

Graduate attributes (GAs) are a required component for accreditation of undergraduate engineering programs in Canada. Similar to ABET student outcomes in the United States, they attempt to quantify a broad range of technical and non-technical skills that engineers are expected to possess upon graduation. However, the heterogenous nature of the data makes it difficult to draw meaningful conclusions using basic statistics, necessitating the use of more advanced analysis. This work presents and applies a mixed-effects model to six years of GA data from the final two years of a four-year chemical engineering program as a means of providing a clearer measure of overall student performance on specific outcomes by separating heterogenous factors such as assessment method and course offering from overall student performance. Through a mixed-effects modeling approach, assessment method was found to have a strong effect on observed performance, with test questions showing highly reduced scores relative to all observations, while presentations and oral examinations demonstrated inflated scores. Course offering was applied as a second effect to account for the specific impact of instructor, course level, and cohort on baseline performance. Although knowledge base was found to be weakest attribute using basic statistics, the mixed-effects model identified that this was largely attributable to the predominant use of more challenging assessments such as exam questions and that communication may be a weaker attribute once assessment difficulty is accounted for. Monitoring GAs using a mixed-effects approach provides a more reliable evaluation of student performance than basic statistics, thereby strengthening continual program improvement. • Graduate attribute data analyzed by mixed-effects model to remove confounding variables • Presentations inflate observed performance; tests decrease observed performance • Communication skills observed to be worst-performing graduate attribute • Improved tracking recommended to account for other variables