Development and Pilot Evaluation of a Standardized and Accessible Rubric (OMNI) for Assessing Generative AI in Health Care Contexts: Four-Stage Mixed Methods Study

Background: The rapid adoption of generative AI in the community and health care necessitates reliable, human-led evaluation. Objective: This study aimed to develop a standardized rubric to support health care professionals, administrators, and researchers in evaluating generative AI outputs in health contexts. Methods: A rubric was developed through a 4-stage mixed methods study: literature synthesis, expert refinement, pilot application, and post-pilot feedback. The rubric was informed by 69 publications and refined through iterative feedback from a multidisciplinary expert panel (n=23). The pilot rubric was applied to 4 health care scenarios: responses to health-related Reddit questions, generation of health-related images and videos, responses to health professional continuing education questions, and automated physician-to-physician clinical documentation generation. Via these scenarios, 15 clinicians (medical practitioners, pharmacists, and medical oncologists) evaluated 250 input-output pairs using domain-level ratings and qualitative comments. The evaluated outputs included 200 AI-generated responses across the 4 health care scenarios and 50 physician-tagged replies in the health-related Reddit questions scenario. Results: The final rubric, OMNI, comprised 10 evaluation domains (Accuracy, Safety, Recognizes Uncertainty, Completeness, Communicates Clearly, Empathetic, Context Aware, Referencing, Follows Instructions, and Avoids Coerciveness) assessed on 4-point Likert scales, supported by detailed instructional and example materials. Via the 4 pilot scenarios, assessors generated 9600 domain-level ratings and 4010 qualitative comments. The mean pairwise agreement for the 4-point Likert groups ranged from 30.8% to 53.4%, increasing to 59.3% to 79.6% when collapsed into broad agreement categories; Gwet's AC2 ranged from 0.49 to 0.80. OMNI differentiated strengths and weaknesses between models within scenarios, while recurrent limitations in Referencing, Recognizes Uncertainty, Completeness, Empathetic, and Accuracy were identified across scenarios. Multimodal image and video outputs had the highest occurrence of negative rating domains. Assessors reported OMNI was usable and indicated they would reuse it. Conclusions: OMNI is a standardized, accessible rubric for expert-led evaluation of generative AI outputs in health contexts. Pilot application demonstrated that OMNI could differentiate AI performance across models and domains while generating structured feedback. Further work is needed to assess its applicability to purpose-built health AI tools, agentic systems, and organizational evaluation settings.

Authors

Institutions

Publication Details

Journal
Journal of Medical Internet Research
Published
2026-10-27
DOI
https://doi.org/10.2196/100531
Primary Topic
Artificial Intelligence in Healthcare and Education
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
article

Development and Pilot Evaluation of a Standardized and Accessible Rubric (OMNI) for Assessing Generative AI in Health Care Contexts: Four-Stage Mixed Methods Study

Stephen Bacchi, Sandeep Reddy, Ganessan Kichenadasse, Paul David Molyneux et al.
4 links
Journal of Medical Internet Research
Artificial Intelligence in Healthcare and Education
article

Development and Pilot Evaluation of a Standardized and Accessible Rubric (OMNI) for Assessing Generative AI in Health Care Contexts: Four-Stage Mixed Methods Study

Stephen Bacchi, Sandeep Reddy, Ganessan Kichenadasse, Paul David Molyneux, Nadia Hitchen, Martin Hong, Adel Shahnam, Huah Shin Ng, Imogen Ramsey, Nicole M. Kuderer, Lee Xingzhuo Li, Amin Rostami‐Hodjegan, Nicholas L. Scarfo, Ross Allan McKinnon, Arduino Aleksander Mangoni, Benjamin Hull Chin-Yee, Warit Ruanglertboon, Ashley Mark Hopkins, Melissa D. McCradden, Michael Joseph Sorich, Ryan Calabro, Sally B. Marotti, Jessica M. Logan, Gary H. Lyman, Jimit Gandhi, Joanne Tang, Bradley D. Menz, James Goonan, Mythily Aravinthan, Ahmad Y. Abuhelwa, Divya Joyce Susai Manickam, Brinleigh Murphy-Reuter, Stephanie Fontanarosa, Erik Cornelisse, David Postma, Kezia Daniel, Dib Kousa, Joy Mariadas, Bianca van Diermen, Hannah Carthew
article en

Abstract

Background: The rapid adoption of generative AI in the community and health care necessitates reliable, human-led evaluation. Objective: This study aimed to develop a standardized rubric to support health care professionals, administrators, and researchers in evaluating generative AI outputs in health contexts. Methods: A rubric was developed through a 4-stage mixed methods study: literature synthesis, expert refinement, pilot application, and post-pilot feedback. The rubric was informed by 69 publications and refined through iterative feedback from a multidisciplinary expert panel (n=23). The pilot rubric was applied to 4 health care scenarios: responses to health-related Reddit questions, generation of health-related images and videos, responses to health professional continuing education questions, and automated physician-to-physician clinical documentation generation. Via these scenarios, 15 clinicians (medical practitioners, pharmacists, and medical oncologists) evaluated 250 input-output pairs using domain-level ratings and qualitative comments. The evaluated outputs included 200 AI-generated responses across the 4 health care scenarios and 50 physician-tagged replies in the health-related Reddit questions scenario. Results: The final rubric, OMNI, comprised 10 evaluation domains (Accuracy, Safety, Recognizes Uncertainty, Completeness, Communicates Clearly, Empathetic, Context Aware, Referencing, Follows Instructions, and Avoids Coerciveness) assessed on 4-point Likert scales, supported by detailed instructional and example materials. Via the 4 pilot scenarios, assessors generated 9600 domain-level ratings and 4010 qualitative comments. The mean pairwise agreement for the 4-point Likert groups ranged from 30.8% to 53.4%, increasing to 59.3% to 79.6% when collapsed into broad agreement categories; Gwet's AC2 ranged from 0.49 to 0.80. OMNI differentiated strengths and weaknesses between models within scenarios, while recurrent limitations in Referencing, Recognizes Uncertainty, Completeness, Empathetic, and Accuracy were identified across scenarios. Multimodal image and video outputs had the highest occurrence of negative rating domains. Assessors reported OMNI was usable and indicated they would reuse it. Conclusions: OMNI is a standardized, accessible rubric for expert-led evaluation of generative AI outputs in health contexts. Pilot application demonstrated that OMNI could differentiate AI performance across models and domains while generating structured feedback. Further work is needed to assess its applicability to purpose-built health AI tools, agentic systems, and organizational evaluation settings.

Journal of Medical Internet ResearchVol. 28
Western University (CA), Prince of Songkla University (TH), Memorial Sloan Kettering Cancer Center (US), Queensland University of Technology (AU), Flinders University (AU), Duke University (US), Peter MacCallum Cancer Centre (AU), Lyell McEwin Hospital (AU), Flinders Medical Centre (AU), University of Sharjah (AE), Fred Hutch Cancer Center (US), Women's and Children's Health Network (AU), Cancer Council SA (AU), SA Health (AU), Adelaide Institute for Sleep Health (AU), Duke Medical Center (US), South Western Sydney Local Health District (AU), Adelaide University (AU), Caring Futures Institute (AU), Sickkids Research Institute (CA), The University of Adelaide (AU)
Good health and well-being
Openalex Percentile: Top 20%
Artificial Intelligence in Healthcare and Education
DOIOpen Access PDFOpenAlex
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon