Beyond the Black Box: A 30-Minute Artificial Intelligence Critical Appraisal Workshop for Internal Medicine Residents

Introduction: Artificial intelligence (AI) and large language models (LLMs) are rapidly entering resident workflows. However, formal training on appropriate LLM use is limited, and AI outputs may be inaccurate, outdated, or biased. This creates a curricular gap: residents may use AI before receiving guidance on evaluating AI-generated recommendations. Methods: We developed a 30-minute AI critical appraisal workshop for postgraduate year 1 to 3 (PGY-1-PGY-3) internal medicine residents. The workshop replaced 1 evidence-based medicine (EBM) journal club session during ambulatory continuity clinic blocks. Six workshops were delivered from July to December 2025. Learners prompted an LLM using a fictional patient scenario, compared outputs in small groups, and learned a structured appraisal framework. Anonymous surveys were collected immediately before and after the workshop using QR (quick-response) codes. Results: Of 72 eligible residents, 53 completed preworkshop surveys and 44 completed postworkshop surveys. Responses were anonymous and unpaired, so results were analyzed descriptively. Confidence critically appraising AI output increased from 60.4% preworkshop to 79.5% postworkshop. Concern about inaccurate AI predictions increased from 69.8% to 88.6%, and concern about AI bias increased from 77.4% to 88.6%. Belief that AI improves health outcomes remained high: 94.3% and 97.7%. Overall, 90.5% rated the session 4 or 5 out of 5. Discussion: A brief, required workshop teaching residents AI literacy and critical appraisal skills was feasible and well received. Postworkshop respondents reported greater confidence in critically appraising AI output and greater concern about inaccurate predictions and bias than preworkshop respondents. The curriculum is adaptable, requires minimal technology, and can be integrated into existing EBM teaching.

Authors

Institutions

Publication Details

Journal
MedEdPORTAL
Published
2026-11-16
DOI
https://doi.org/10.15766/mep_2374-8265.11654
Primary Topic
Artificial Intelligence in Healthcare and Education
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

Beyond the Black Box: A 30-Minute Artificial Intelligence Critical Appraisal Workshop for Internal Medicine Residents

Joshua I. Daum, Christopher R. Stephenson, Carrie A. Thompson, Brianna E. Vaa Stelling et al.
3 links
MedEdPORTAL
Artificial Intelligence in Healthcare and Education
article

Beyond the Black Box: A 30-Minute Artificial Intelligence Critical Appraisal Workshop for Internal Medicine Residents

Joshua I. Daum, Christopher R. Stephenson, Carrie A. Thompson, Brianna E. Vaa Stelling, Kelsey Lynch
article en

Abstract

Introduction: Artificial intelligence (AI) and large language models (LLMs) are rapidly entering resident workflows. However, formal training on appropriate LLM use is limited, and AI outputs may be inaccurate, outdated, or biased. This creates a curricular gap: residents may use AI before receiving guidance on evaluating AI-generated recommendations. Methods: We developed a 30-minute AI critical appraisal workshop for postgraduate year 1 to 3 (PGY-1-PGY-3) internal medicine residents. The workshop replaced 1 evidence-based medicine (EBM) journal club session during ambulatory continuity clinic blocks. Six workshops were delivered from July to December 2025. Learners prompted an LLM using a fictional patient scenario, compared outputs in small groups, and learned a structured appraisal framework. Anonymous surveys were collected immediately before and after the workshop using QR (quick-response) codes. Results: Of 72 eligible residents, 53 completed preworkshop surveys and 44 completed postworkshop surveys. Responses were anonymous and unpaired, so results were analyzed descriptively. Confidence critically appraising AI output increased from 60.4% preworkshop to 79.5% postworkshop. Concern about inaccurate AI predictions increased from 69.8% to 88.6%, and concern about AI bias increased from 77.4% to 88.6%. Belief that AI improves health outcomes remained high: 94.3% and 97.7%. Overall, 90.5% rated the session 4 or 5 out of 5. Discussion: A brief, required workshop teaching residents AI literacy and critical appraisal skills was feasible and well received. Postworkshop respondents reported greater confidence in critically appraising AI output and greater concern about inaccurate predictions and bias than preworkshop respondents. The curriculum is adaptable, requires minimal technology, and can be integrated into existing EBM teaching.

MedEdPORTALVol. 22
Mayo Clinic (US), Mayo Clinic in Arizona (US), Mayo Clinic in Florida (US)
Good health and well-being, Quality education
Artificial Intelligence in Healthcare and Education
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon