Exploring the Impact of Unsupervised Clustering Methods in Systematic Literature Reviews

About 15 years ago, education researchers conducted a systematic literature review (SLR) on change strategies for improving undergraduate STEM education instruction. Analyzing 191 articles from 1995 to 2008, the researchers identified four broad categories of change strategies through a comprehensive interdisciplinary literature review: (1) disseminating curriculum and pedagogy, (2) developing reflective teachers, (3) enacting policy, and (4) developing a shared vision. With recent developments in STEM education practices, it is imperative to conduct a follow-up SLR comparing the effects of change strategies and associated student success. Similarly, with the influx of scientific articles published in recent decades, it is time consuming to conduct comprehensive SLRs without the assistance of machine learning (ML) analysis techniques. Working with qualitative researchers, we investigate the impact and ability of using ML to assist in these analyses. While most ML approaches can be easily written with a few lines of code, using them to extract meaningful information for literature reviews is challenging. In this work, we describe our experience with integrating machine learning techniques into the analysis pipeline of SLRs. Specifically, this poster will: (1) share results from clustering analysis to identify themes of the chosen abstracts, (2) explore the effects of data bias on found clusters, (3) present the challenges of adding machine learning into SLRs, and (4) assess if the addition of ML in SLRs aligns with the expected goals of qualitative researchers.