A robust vision language model for molecular status prediction and radiology report generation in adult-type diffuse gliomas

We aimed to establish a robust vision-language model ("Glio-LLaMA-Vision") for molecular status prediction and radiology report generation (RRG) in adult-type diffuse gliomas. Multiparametric MRI data and paired radiology reports from 1001 patients with adult-type diffuse gliomas were included in the institutional training set. A vision-language model, Glio-LLaMA-Vision, was developed from LLaMA 3.1 pre-trained on 2.79 million biomedical image-text pairs from PubMed Central and further fine-tuned from the institutional training set. The performance was validated in 100 patients and 75 patients with paired MRI-radiology reports from an institutional validation set and another tertiary institution (AMC), and in 170 and 477 patients with MRI from TCGA and UCSF datasets, respectively. In terms of IDH mutation status prediction, Glio-LLaMA-Vision showed AUCs ranging from 0.85-0.95 in the internal validation and external datasets. In terms of RRG, the BLEU-1 and ROUGE-L scores were 0.50 and 0.49 in the internal validation, respectively, and 0.32 and 0.36 on the AMC dataset, respectively. Overall, 37.8% of generated reports were considered superior or equal to the original reports, while 91.0% of generated reports were considered clinically acceptable by neuroradiologists. In conclusion, Glio-LLaMA-Vision demonstrates promising performance in molecular status prediction and RRG in adult-type diffuse gliomas, showing potential for clinical assistance.