Assessing the quality of AI-assisted peer reviews: A comparative assessment of ChatGPT, Claude Sonnet and Google Gemini in reviewing scientific articles
Objectives: Peer review validates research findings and upholds the quality and credibility of scientific publications. This study aimed to assess the quality of peer-review reports generated by GPT-5.6, Claude Sonnet-4.6, and Google Gemini-3.1 Pro, while evaluating the same set of published scientific articles. Methodology: This comparative study was performed in the “Department of Physiology, College of Medicine, King Saud University, Riyadh, Saudi Arabia” from August 23, 2026, to September 10, 2026. In this study, 30 original published articles were identified, and each article was submitted to GPT-5.6 (n=30), Claude Sonnet-4.6 (n=30), and Google Gemini-3.1 Pro (n=30). Each AI model generated 30 individual review reports, for a total of 90 peer-review reports. A panel of five faculty members evaluated the peer-review reports, and each report was scored out of five marks. Each examiner evaluated peer-review reports generated by ChatGPT (30), Claude Sonnet (30), and Google Gemini (30). The frequencies, percentages, mean ± SEM and 95% CIs were calculated, and p-values < 0.05 were considered statistically significant.Results: AI models generated 90 peer-review reports, 30 each from ChatGPT, Claude Sonnet, and Google Gemini. Across the five examiners, ChatGPT received an overall grade of 634.1/750 (84.54%), Claude Sonnet received 690.4/750 (92.05%), and Google Gemini received 465.3/750 (62.04%). Claude’s Sonnet review reports received the highest grades, followed by ChatGPT and Google Gemini (p=0.001). Conclusions: The AI model, Claude Sonnet-4.6, produces significantly better review reports than GPT-5.6 and Google Gemini-3.1 Pro. These AI models summarize complex information and identify potential gaps, methodological concerns, and issues in the results and discussion sections of medical science articles. Integrating AI models within professional boundaries supports academic journals’ editorial workflows rather than replacing human peer review.
Authors
- Muhammad Zain Sultan Meo
- Riham A. ElToukhy (ORCID: https://orcid.org/0009-0009-6475-7903)
- Sultan Ayoub Meo (ORCID: https://orcid.org/0000-0001-9820-1852)
- Yasir Waheed (ORCID: https://orcid.org/0000-0002-5789-4215)
- David Charles Klonoff (ORCID: https://orcid.org/0000-0001-6394-6862)
- Naseer Ahmed (ORCID: https://orcid.org/0000-0002-0960-1123)
Institutions
- Sutter Health (US)
- Alfaisal University (SA)
- University of California, San Francisco (US)
- King Saud University (SA)
- Mills Peninsula Health Services (US)
- Rehman Medical Institute (PK)
Publication Details
- Journal
- Pakistan Journal of Medical Sciences
- Published
- 2026-11-19
- DOI
- https://doi.org/10.12669/pjms.42.12.21007
- Primary Topic
- Artificial Intelligence in Healthcare and Education
- Type
- article
- Field-Weighted Citation Impact
- 0.00