Prompting to Bridge the Gap: Accelerating Research in Multi-Modal Medical Report Generation for Alzheimer’s Disease

Research on Medical Report Generation (MRG) aims to make medical information more accessible to patients by generating lucid and medically accurate reports. To generate reports reliably, current foundation models require considerable fine-tuning. However, creating relevant datasets demands significant time and effort from experts. This raises the need for methodologies to accelerate the development of MRG data and methods for disease diagnoses that have not been studied previously. This motivates us to propose, as a first work of its kind, a low-resource Cognition-Guided Prompting for Alzheimer’s disease MRG (CGP-MRG) using multi-modal patient data. CGP-MRG first predicts patient cognitive status from input modalities, which is then incorporated into a zero-shot prompt for a fine-tuned LLM. To make possible the fine-tuning of this LLM, we also propose an LLM-assisted prompt engineering framework, which progressively learns the structure of a prompt composed of multiple candidate sentences, with low-effort expert guidance from physicians for medical accuracy, along with low-effort guidance by non-experts for language clarity and variability. The learned prompt lets us generate a dataset containing 20,000 medical reports corresponding to different stages of cognitive health. For CGP-MRG, we also curated a multi-modal dataset from the ADNI to train a subnetwork that predicts patient cognitive status. Empirical studies show that CGP-MRG, especially with a fine-tuned Llama 3, excels in Alzheimer’s disease MRG across several metrics. Our methodology demonstrates how LLMs can aid in the rapid development of MRG methods for a previously unstudied disease, even when there is a complete absence of previous MRG data.