Pandora: an Ai Model for the Automatic Extraction of Clinical Unstructured Data and Clinical Risk Score Implementation

Introduction: Medical records and physician notes contain valuable non-tabular information that requires significant manual effort to extract and structure. Large Language Models (LLMs) have demonstrated the ability to understand, reason, and retrieve information from such sources, transforming it into accessible information for clinical or research purposes. Objective: To present and asses the capabilities of PANDORA, our AI system, comprised of two LLMs that can extract data and use it in validated calculator and prediction models to provide recommendations. Methods: The study evaluates the model's ability to extract clinical features from discharge notes from the MIMIC database and synthetically generated outpatient charts. We use the PUMA calculator for Chronic Obstructive Pulmonary Disease (COPD) case finding, which interacts with the model to calculate a score based on seven criteria and determine which patients should undergo further spirometry testing. Results: The model's exhibited excellent extraction accuracy, achieving 100 % for MIMIC and 99 % for synthetic cases. Interaction with the PUMA scale produced accurate scores, with an accuracy of 94% for both databases. The final recommendation on COPD risk was based on the PUMA scale, classified as positive if score <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$\geq 5$</tex>. Sensitivity was 86 % for MIMIC and 100 % for synthetic cases. Conclusion: LLMs have been successful in extracting information and generating recommendations. However, this is the first model that successfully extracts information based on existing, validated clinical scores from plain, non-tabular data and provides a recommendation mixing all these capabilities. It leverages existing knowledge, making it available to be explored in light of the highest-quality evidence in several medical fields.