Analysis of Student Output on the Use of ChatGPT: A Predictive Model Approach

Artificial Intelligence (AI) has significantly transformed various aspects of education, with AI-powered language models like ChatGPT gaining popularity due to their unique features and advantages. This study aims to analyze student outputs and develop a predictive model to assess whether essay-type answers, Dropbox submissions, and machine problems were generated using ChatGPT, employing machine learning algorithms such as Naive Bayes (NB), Random Forest (RF), and K-Nearest Neighbors (KNN). Student outputs are evaluated using six AI detection tools: Contentatscale, Crossplag, GPTZero, KazanSEO, Sapling, and ZeroGPT. The results are predicted by NB, RF, and KNN, which were chosen for their strong performance in text classification, robustness, and ability to manage non-linear data. The analysis examines performance metrics, including Recall, Precision-Recall Curve (PRC) Area, and Class Accuracy, to provide insights into the predictive capabilities of these models. The findings reveal that NB outperformed the other algorithms, achieving the highest correctly classified instances at 23.19% and a Kappa statistic of 0.1072, indicating slight agreement in classification accuracy, while RF and KNN recorded 14.49% and 15.94%, respectively. Additionally, NB demonstrated the highest true positive rate of 0.232 and PRC area of 0.466, while KNN achieved the best PRC area at 0.566, reflecting varied performance across models. Generally, while Naive Bayes showed superior accuracy and predictive ability, each model has unique strengths that can be leveraged to analyze student outputs and evaluate the use of tools like ChatGPT in educational settings.