Clinical Verification of AI Models for Radiology Report Annotation Using a Physician-Annotated MIMIC-CXR

Accurate evaluation of AI-generated radiology reports is crucial for clinical applications. This study investigates the use of large language models (LLMs) - specifically Google Gemini and OpenAI GPT-4o - for this task. We compare their performance against CheXpert in annotating both raw radiology reports and reports generated by image-to-report systems (MedRAX, RGRG), using a physician-validated subset of MIMIC-CXR as ground truth. Our evaluation demonstrates Gemini’s strong performance in this context. This work provides a clinical validation of LLMs against expert annotations, highlighting their potential for medical text processing and the challenges in accurately identifying rare findings.