Using a Diverse Test Suite to Assess Large Language Models on Fast Health Care Interoperability Resources Knowledge: Comparative Analysis
This study highlights the competitive performance of both open-source models, such as Qwen and DeepSeek, and commercial models, such as GPT-4o and Gemini, in FHIR-related tasks. While open-source models are advancing rapidly, commercial models still have an advantage for specific, complex tasks. The FHIR Workbench offers a valuable platform for evaluating the capabilities of these models and promoting improvements in health data interoperability.
Authors
- René Hosch (ORCID: https://orcid.org/0000-0003-1760-2342)
- Henning Schäfer (ORCID: https://orcid.org/0000-0002-4123-0406)
- Johannes Haubold (ORCID: https://orcid.org/0000-0003-4843-5911)
- Eva Maria Hartmann (ORCID: https://orcid.org/0009-0000-2600-7217)
- Cynthia Sabrina Schmidt (ORCID: https://orcid.org/0000-0003-1994-0687)
- Obioma Pelka (ORCID: https://orcid.org/0000-0001-5156-4429)
- Christoph M. Friedrich (ORCID: https://orcid.org/0000-0001-7906-0038)
- Mikel Bahn
- Katarzyna Borys (ORCID: https://orcid.org/0000-0001-6987-6041)
- Ahmad Idrissi-Yaghir (ORCID: https://orcid.org/0000-0003-1507-9690)
- Felix Nensa (ORCID: https://orcid.org/0000-0002-5811-7100)
- Kamyar Arzideh (ORCID: https://orcid.org/0009-0005-6074-804X)
- Bahadır Eryılmaz (ORCID: https://orcid.org/0009-0002-8743-4751)
- Yutong Wen (ORCID: https://orcid.org/0009-0003-8557-6665)
Institutions
- Dortmund University of Applied Sciences and Arts (DE)
- Institut für Medizinische Informatik, Biometrie und Epidemiologie (DE)
- Ruhrlandklinik (DE)
- Essen University Hospital (DE)
Publication Details
- Journal
- Journal of Medical Internet Research
- Published
- 2025-08-12
- DOI
- https://doi.org/10.2196/73540
- Citations
- 1
- Primary Topic
- Artificial Intelligence in Healthcare and Education
- Type
- article
- Field-Weighted Citation Impact
- 0.65