Large language model assistance for report-based carotid plaque AHA classification: a multi-reader study

OBJECTIVES: To evaluate whether large language model assistance improves radiologists' agreement with a text-based reference standard in inferring the modified American Heart Association carotid plaque classification from free-text MRI reports, and whether a lightweight model can perform well under criteria-guided prompting. MATERIALS AND METHODS: This retrospective study included 433 patients from three institutions. In a report-based reader study, four radiologists classified MRI reports with and without large language model assistance. In standalone evaluation, a rule-based baseline and 10 models were compared under naive and criteria-guided prompting. The reference standard was expert consensus based on the report text. RESULTS: Among 433 patients (mean age, 65.9 years ± 13.4; 248 males), AI assistance improved six-class agreement with the text-based reference standard. At the vessel level, junior radiologists improved from 61.75% to 83.12% (Qwen3-8B) and 87.75% (DeepSeek-R1); senior radiologists from 77.00% to 87.50% and 92.25%, respectively (all Holm-corrected p < 0.001). Patient-level agreement increased from 41.0% to 78.8% (junior) and from 59.8% to 87.8% (senior) with DeepSeek-R1. Agreement increased (quadratic-weighted κ: 0.231 without AI vs 0.775 with DeepSeek-R1). In standalone evaluation, criteria-guided prompting improved Qwen3-8B agreement from 38.91% to 81.52% (95% CI: 78.87-84.06%), remaining 5.89 percentage points below DeepSeek-R1 (87.41%; 95% CI: 85.10-89.61%). CONCLUSION: In this report-based classification setting, large language model assistance improved radiologists' agreement with the text-based reference standard, with larger gains in junior radiologists. When prompts explicitly encoded AHA criteria, an 8B local candidate model performed well, supporting its potential use as a standardization aid for text-based workflows. KEY POINTS: Question Report-based modified AHA classification of carotid plaques is experience-dependent and difficult to standardize, as mapping multi-sequence imaging findings to classification conclusions imposes substantial cognitive load. Findings Large language model assistance improved agreement with a text-based reference standard and raised inter-reader consistency (weighted κ 0.231-0.775), narrowing the junior-senior gap. Critical relevance statement Embedding explicit AHA criteria in the prompt let an 8B model approach 671B-model agreement (38.91-81.52%), supporting standardized text-based workflows without fine-tuning.