P011 Automated detection of in-hospital drug hypersensitivity reactions using a privacy-preserving large language model

Abstract Drug hypersensitivity reactions (DHRs) are a significant clinical concern, leading to severe complications, prolonged hospital stays and increased healthcare costs. Despite their importance, DHRs are under-reported, limiting the ability to estimate their true prevalence and impeding effective pharmacovigilance. Current methods for detecting DHRs, such as spontaneous reporting systems and manual reviews, are either insufficient or labour intensive, emphasizing the need for automated approaches. This study evaluated Llama 3.1 (70b), an open-source large language model (LLM), for detecting allergic adverse drug events in hospital discharge summaries. In total 305 randomly selected French discharge summaries were analysed, with parameters set to achieve 90–95% sensitivity for DHR detection. Llama 3.1 (70b) identified the presence of DHRs, the causative drug, the allergic mechanism (immediate or delayed) and pharmacovigilance reporting. Four board-certified dermatologists, blinded to each other’s evaluations and the model’s output, reviewed the summaries to create a consensus-based gold standard. Out of 305 summaries, 27 DHRs were identified, primarily delayed reactions (70%). Notably, none of these were reported to pharmacovigilance agencies. While dermatologists required 2.5–4 h for review, Llama 3.1 completed the task in 17 min. The model achieved 85% sensitivity and 84% specificity, closely mirroring expert performance. Interestingly, all DHRs missed by Llama were also overlooked by at least one expert. Llama 3.1 demonstrated strong accuracies in identifying the responsible drug (88%), the allergic mechanism (96%) and pharmacovigilance reporting (88%). Its performance underscores the potential of LLMs as efficient, cost-effective tools for large-scale DHR detection. The study also highlighted the absence of prompt engineering, reflecting real-world application scenarios, and emphasized the model’s open-source nature, which ensures privacy and scalability. These findings suggest that integrating LLMs like Llama 3.1 into clinical workflows could enhance drug safety surveillance and streamline pharmacovigilance processes. By automating DHR detection, healthcare systems could improve reporting, enhance patient safety, and optimize prescribing practices. Future studies should explore scalability to larger datasets and integration into clinical settings. In conclusion, this study demonstrates the feasibility of using privacy-preserving LLMs for DHR detection, paving the way for improved pharmacovigilance and healthcare practices.