Evaluating open-source LLMs for dental EMR generation

Electronic medical records impose significant documentation burdens. While proprietary Large Language Models (LLMs) offer automation, they raise data sovereignty and privacy concerns. This study evaluated the feasibility of using locally deployable open-source LLMs (specifically DeepSeek-V3) versus proprietary models for dental EMR generation. A dataset of 190 de-identified dental EMRs was converted into synthetic Mandarin doctor-patient dialogues, serving as a high-context linguistic stress test. Six LLMs—two open-source (DeepSeek-V3, LLaMA-3.1-405B) and four proprietary (GPT-4, GPT-4o, Claude-3.7-Sonnet, Grok-3)—were evaluated using the structured “CRISPE Framework” and a baseline “Basic Prompt”. Assessment utilized a dual-track framework: automated metrics (n = 190) and blinded expert clinical evaluation using a 7-Dimensional Index (7DI) on a stratified subset (n = 50). The CRISPE strategy significantly outperformed the basic prompt across all models (Mean 7DI: 28.86 ± 2.12 vs. 26.91 ± 2.36; p<.001). Under optimized prompting, DeepSeek-V3 achieved a mean 7DI score of 28.76 ± 2.04, and we did not detect statistically significant differences versus GPT-4o (29.22 ± 2.25; p>.05) in this expert-rated sample. One-way ANOVA similarly did not detect overall between-model differences (F = 0.54, p=.744); however, non-significance should not be interpreted as equivalence, and larger prospective evaluations are warranted. Structured prompt engineering is critical for enhancing AI-generated documentation. Open-source models demonstrated clinical performance comparable to proprietary leaders when optimally prompted, though larger validation studies are needed. These preliminary findings suggest the feasibility of privacy-preserving local deployment strategies, offering a potential pathway to democratize AI support in dental institutions without compromising data sovereignty.