A Reproducible Decomposed Large Language Model Pipeline for Structured Abstraction of Multimodal Neuroimaging Reports: Development and Independent Validation in 1,669 Emergency Department Encounters for Dizziness and Vertigo

Objective: To develop and independently validate a decomposed, modality-specific large language model (LLM) pipeline that abstracts structured variables from the free-text neuroimaging reports of a multimodal stroke workup, and to validate the resulting structured dataset for reuse in downstream research. Materials and Methods: A frozen pipeline (temperature 0) extracting 22 variables across CT head, CT angiography, CT perfusion, and MRI brain was applied to 1,669 ED encounters for dizziness or vertigo; 11 variables were pre-specified as primary (Tier 1). Reproducibility used three same-session runs and a nine-week repeat run; three frontier models (Claude Sonnet 4.6, GPT-5.4, Gemini 3.5 Flash) measured cross-model agreement (Cohen's/Fleiss' kappa). Accuracy was assessed in an independent 200-encounter cohort (150 random, 50 enriched for rare findings); 100 were double-read by two blinded reviewers with discordances adjudicated. Results: Test-retest reproducibility was high (Tier 1 Fleiss' kappa = 0.993; 99.8% exact) and stable nine weeks later (kappa = 0.992); inter-model agreement was kappa = 0.962. Against the adjudicated reference, pooled Tier 1 accuracy was 0.971 to 0.988 across models (Cohen's kappa 0.959 to 0.982), comparable to inter-reviewer agreement (kappa = 0.932). Accuracy was 0.980 to 0.983 in random and 0.955 to 0.977 in enriched encounters (agreement on model-flagged findings, not sensitivity). Discussion: A decomposed LLM pipeline produced reproducible, model-portable, stable, and accurate multimodal abstraction. Because schema design and validation are up-front costs, such pipelines scale to larger research cohorts. Conclusion: The quantified per-field error rates support reuse of this dataset with field-specific review, and such pipelines offer a practical path to large-scale, reproducible research datasets.