GPT as a Reviewer: Automatic Evaluation of Academic Papers

This study investigates the potential of using GPT models, specifically GPT-3.5 and GPT-4 variants, as automated reviewers in academic peer review processes. Experiments were conducted using the ACL-2017 dataset, employing both zeroshot learning and in-context learning techniques across various settings, including baseline, importance assignment, and persona assignment, with different prompt designs. These settings tested the models’ effectiveness in scoring based on predefined evaluation criteria, both with and without scoring thresholds. The results highlight how various prompt strategies, settings, and threshold applications influenced model performance. Among the models, GPT-40 and GPT-40-mini showed particularly promising results. While GPT models performed well in certain areas, they still have limitations in fully capturing the complexities of peer review. Nevertheless, the findings suggest that GPT models can serve as a helpful tool to support human reviewers in the peer review process.