Unlocking Potential of Generative Large Language Models for Adverse Drug Reaction Relation Prediction in Discharge Summaries: Analysis and Strategy

We present a comparative analysis of generative large language models (LLMs) for predicting causal relationships between drugs and adverse events found in text segments from discharge summaries. Despite lacking prior training for identifying related drug-adverse event pairs, generative LLMs demonstrate exceptional performance as recall-optimized models, achieving F1 scores comparable to those of fine-tuned models. Notably, on the MIMIC-Unrestricted dataset, Gemini 1.5 Pro and Llama 3.1 405B outperform our in-house fine-tuned BioM-ELECTRA-Large, with Gemini 1.5 Pro showing a 19.2% (0.724-0.863) improvement in F1 score and a 39.7% (0.675-0.943) increase in recall, while Llama 3.1 405B exhibits a 12.4% (0.724-0.814) improvement in F1 and a 40.4% (0.675-0.948) boost in recall. Additionally, we propose a hybrid approach that integrates BioM-ELECTRA-Large with generative LLMs, resulting in enhanced performance over the individual models. Our hybrid model achieves F1 score improvements ranging from 0.8% to 18.5% (0.005-0.133) over BioM-ELECTRA-Large in the validation set, primarily due to increased precision, albeit with a decrease in recall compared with the original generative LLM. Importantly, this approach yields substantial computational resource savings, as BioM-ELECTRA-Large selects only a subset of segments-ranging from 19.7% to 73.4% across our datasets-for downstream prediction by generative LLMs. By harnessing the strengths of generative LLMs as recall-optimized models and combining them with fine-tuned models, we can unlock the full potential of artificial intelligence in predicting adverse drug reaction relations, ultimately enhancing patient safety.