Using LLMs to Improve Reproducibility of Literature Reviews

Literature reviews play a crucial role in Information Systems (IS) research. However, scholars have expressed concerns regarding the reproducibility of their results and the quality of documentation. The involvement of human reproducers in these reviews is often hindered by the time-consuming nature of the procedures. The emergence of Large Language Models (LLMs) seems promising to support researchers and to enhance reproducibility. To explore this potential, we conducted experiments using various LLMs, focusing on abstract scanning, and have presented initial evidence suggesting that the application of LLMs in structured literature reviews could assist researchers in refining and formulating rules for abstract scanning. Based on our preliminary findings, we identify potential future research directions in this research in progress paper.