Multi-agent AI system for radiology report structuring and quality assurance with independent radiologist evaluation

Purpose To develop and evaluate a locally deployed multi-agent artificial intelligence (AI) system for radiology report structuring and quality assurance. Materials and methods This retrospective study included 638 radiology reports from computed tomography (CT) examinations of the chest, abdomen, and pelvis dictated by 15 board-certified radiologists in 2023 and 2024. A multi-agent AI system was developed to perform report structuring and quality assurance (QA). The system structured the report into standardized anatomical sections at the sentence level using regular expression (regex) rules and local large language models (LLMs). It also detected mismatches between the Findings and Impression sections, or within sections; gender–anatomy conflicts; and undocumented communication of critical findings. Two board-certified radiologists independently evaluated a 45-report subset, intentionally enriched for system-flagged reports (41 flagged and 4 unflagged). Results The multi-agent AI system structured the Findings sections of all reports (22,270 sentences) into a predefined anatomical format while retaining the original report content. The system flagged 90 (14.1%) reports, most commonly for section mismatches (80 reports, 12.5%). In the radiologist evaluation, both reviewers agreed that 31 (69%) were correctly restructured, 3 reports (7%) were incorrectly restructured, and disagreed on the remaining 11 reports (24%). Both reviewers agreed that no clinically important information was omitted and no fabricated content was introduced. Overall QA performance was rated as “excellent” or “good” in 84% of the evaluated reports, with the remaining reports rated as “fair”. Conclusion A locally deployed multi-agent AI system combined radiology report structuring and QA within a single workflow. The findings support the feasibility of this approach. With further validation, such systems may support standardized reporting and QA in radiology practice.

Authors

Institutions

Publication Details

Journal
Frontiers in Radiology
Published
2026-11-16
DOI
https://doi.org/10.3389/fradi.2026.1961438
Primary Topic
Artificial Intelligence in Healthcare and Education
Type
article
Field-Weighted Citation Impact
0.00

Funders

Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

Multi-agent AI system for radiology report structuring and quality assurance with independent radiologist evaluation

Robert A. Gatenby, Ghulam Rasool, Aliya Qayyum, Christopher Otteni et al.
Frontiers in Radiology
Artificial Intelligence in Healthcare and Education
article

Multi-agent AI system for radiology report structuring and quality assurance with independent radiologist evaluation

Robert A. Gatenby, Ghulam Rasool, Aliya Qayyum, Christopher Otteni, Iryna Hartsock, Cesar A. Lam, Cyrillo Araujo
article en

Abstract

Purpose To develop and evaluate a locally deployed multi-agent artificial intelligence (AI) system for radiology report structuring and quality assurance. Materials and methods This retrospective study included 638 radiology reports from computed tomography (CT) examinations of the chest, abdomen, and pelvis dictated by 15 board-certified radiologists in 2023 and 2024. A multi-agent AI system was developed to perform report structuring and quality assurance (QA). The system structured the report into standardized anatomical sections at the sentence level using regular expression (regex) rules and local large language models (LLMs). It also detected mismatches between the Findings and Impression sections, or within sections; gender–anatomy conflicts; and undocumented communication of critical findings. Two board-certified radiologists independently evaluated a 45-report subset, intentionally enriched for system-flagged reports (41 flagged and 4 unflagged). Results The multi-agent AI system structured the Findings sections of all reports (22,270 sentences) into a predefined anatomical format while retaining the original report content. The system flagged 90 (14.1%) reports, most commonly for section mismatches (80 reports, 12.5%). In the radiologist evaluation, both reviewers agreed that 31 (69%) were correctly restructured, 3 reports (7%) were incorrectly restructured, and disagreed on the remaining 11 reports (24%). Both reviewers agreed that no clinically important information was omitted and no fabricated content was introduced. Overall QA performance was rated as “excellent” or “good” in 84% of the evaluated reports, with the remaining reports rated as “fair”. Conclusion A locally deployed multi-agent AI system combined radiology report structuring and QA within a single workflow. The findings support the feasibility of this approach. With further validation, such systems may support standardized reporting and QA in radiology practice.

Frontiers in RadiologyVol. 6
Moffitt Cancer Center (US)
H. Lee Moffitt Cancer Center and Research Institute
Good health and well-being
Artificial Intelligence in Healthcare and Education
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon