Toward Human-Aligned LLM Reviews for Scientific Papers

The peer review process is strained by increasing submission volumes, reviewer fatigue, and inconsistent standards. While Large Language Models (LLMs) can aid in reviews, they are often overly optimistic and lack technical depth. We developed an innovative prompting strategy that, when applied to ChatGPT-4 on ICLR 2025 papers, reduced score inflation and generated reviews more closely aligned with human reviewer median scores.