S3004 Who's Reading Your Clinical Documentation? Assessing Litigation Risk in the Age of AI

Introduction: Medical malpractice claims arise from alleged deviations from standard care. With large language models (LLMs) widely available, patients now possess unprecedented tools to analyze clinical documentation. This raises questions regarding how LLMs might empower patients to interpret care quality and potentially influence malpractice litigation. We designed a novel study exploring this intersection of patient agency, medicolegal systems, and artificial intelligence. Methods: We selected 6 LLMs (Chat GPT 4o, Google Gemini 2.0, Claude 3.7, Meta AI, Copilot, and Grok 3). We benchmarked their understanding of 5 American College of Gastroenterology guidelines. We created 5 fictional patient cases with varied outcomes and illness severity, aligning with societal guidelines and standard care, plus control cases demonstrating substandard care. A multidisciplinary panel reviewed cases, followed by structured queries prompting LLMs to identify deviations from US standards of care. We also emulated layperson queries regarding care quality and litigation, performing thematic and quantitative analyses. Results: All LLMs passed baseline assessments. In standard-care scenarios, Copilot, Gemini, and Claude incorrectly indicated substandard care in 1/5, 2/5, and 1/5 cases, respectively. Gemini and Copilot directly criticized care; Gemini suggested more aggressive resuscitation in suspected variceal bleeding. Claude highlighted omitted details, like thiamine dosages in alcohol withdrawal. Chat GPT, Meta AI, and Grok correctly identified all standard of care cases. For litigation advice, Gemini used strong phrasing (“Factors suggesting malpractice”); Claude and Copilot were more nuanced. All models recommended expert legal consultation. In the control scenarios, all correctly identified substandard care. Meta AI was most effective (17/19 issues), Copilot least (9/19). LLMs consistently flagged major care failures (e.g., emergency surgery delays) as litigation potential but did not directly recommend legal action without expert input. Conclusion: This study shows some LLMs misclassify guideline-adherent care as substandard and occasionally miss genuine care deficiencies. These inaccuracies highlight ongoing needs for oversight and expert validation in medicolegal contexts, emphasizing careful provider documentation in the AI era.