The Essential Role of Pediatric Trainees in Developing Ethical Artificial Intelligence Tools

Dr Atrix, a pediatric resident covering the wards overnight in an underserved community, is called to examine a 1-year-old child with fever and lethargy. The hospital recently implemented an artificial intelligence (AI) sepsis risk stratification calculator to aid in recognition of serious bacterial infection and support escalation decisions. The AI sepsis risk calculator assigned a low-risk score to the patient based on previously recorded data in the patient’s electronic medical record, current examination findings, and vitals. Dr Atrix feels conflicted. The patient’s perfusion is poor, and the family describes rapidly worsening symptoms. Trusting her training, Dr Atrix escalates the case and orders blood cultures. Hours later, the patient becomes septic. Surprised by the AI sepsis risk calculator’s low-risk score, Dr Atrix brings the discordance to her attending physician. With their guidance and support, she follows the institutional pathway to contact the information technology team involved with model implementation for an explanation. After multiple emails, she is provided more information on the model and learns it has not been retrained on recent cases or adjusted for underrepresented subpopulations like non-English speaking immigrant families, who often present later into their course of illness owing to access barriers. These characteristics make up a significant portion of the patients that she sees.AI-enabled clinical decision support tools, such as risk score calculators, triage tools, and radiology aids, are no longer theoretical. They have arrived. How we implement and maintain their use in children’s hospitals will significantly impact equity of care. As trainees, we will be among the first and most frequent users of these tools. This trainee-authored perspective reflects the clinical reality that residents are often early, high-contact users of new digital tools. We must be prepared not just to use AI but to shape it. In an era of rapid uptake of new technologies in health care, we must insist that AI in the pediatric population is designed and governed ethically for our patients. Recent reviews document AI-enabled clinical decision support tools already embedded in emergency triage, radiology, and diagnostic workflows across children’s hospitals nationwide, with pediatric-specific applications expanding rapidly.1,2The risks for inequitable care are uniquely high in pediatrics, and the perpetuation of existing health disparities with AI tools is an unacceptable outcome.3 Pediatric patients rely on proxy decision-makers with varying levels of digital literacy to make decisions on their behalf.Pediatric datasets are also less robust owing to low sample size and demographic heterogeneity.2,4 Because of the developmental and physiologic heterogeneity of pediatric patients, training datasets often fails to adequately represent the full range of ages, stages, and presentations, generating an age-related algorithmic bias. Additional sources of bias include the protected research population status of children, which limits data availability and the limited translatability of adult AI workflows to pediatric settings, particularly where adolescent confidentiality must be preserved.5 Data gaps and biased training of AI models systematically worsen care for children from minoritized racial or ethnic groups, rural communities, low-income families, and non-English speaking households.6,7 Conversely, preventable morbidity and mortality may occur if there are extensive delays in implementing validated pediatric AI tools.8 Existing ethical frameworks, such as the World Health Organization’s (WHO) guidance on AI for health and the United National Children’s Fund’s (UNICEF) guidance on AI for children, provide a potential foundation for AI use in our patient population.9,10Trainees are high-contact users at the bottom of a hierarchical professional system. As the clinicians ordering the majority of tests, write notes, and work in units across the hospital and clinic systems, trainees will have a high volume of interaction with AI tools. We will be the first to notice when AI outputs fail to capture the correct data or provide appropriate instructions. Hospital systems must make trainees feel safe questioning tools that leadership has endorsed. Any pediatric AI governance model that does not include trainee voice will be incomplete and potentially blind to important errors. Unlike rule-based or algorithm-driven clinical decision support tools, machine learning-based AI models may behave as black boxes, with outputs that are difficult to interrogate even by those deploying them. They can be rapidly deployed at scale and may embed training data biases that are not immediately visible to clinicians at the point of care.1 This engagement is most effective when embedded within formal reporting structures and mentored by experienced clinical faculty, ensuring that trainee observations are contextualized alongside clinical expertise without being positioned as independent arbiters of tool appropriateness. Although trainees are the focus of this piece, responsibility for safe and ethical AI use is shared across the clinical team and extends beyond academic medical centers to community hospitals and outpatient settings where trainee involvement may be limited.Pediatric trainees can participate in the development and governance of AI tools at micro-, meso-, and macrolevels (Supplemental Figure), with each level representing a distinct arena for advocacy, education, and accountability.Microlevel actions represent advocacy at the bedside and on the wards. Implementing an AI Safety Checklist can mandate transparency regarding tool development: On what population was this model trained and validated?Were pediatric patients excluded by age, diagnosis, race or ethnicity, language, and/or disability status?Is there an escalation path when the tools seem to be performing poorly?How are families informed that AI is being used in their care?Trainees should document and report concerning patterns through a formalized system, in collaboration with supervising attendings and quality improvement (QI) teams, to ensure that observations are appropriately contextualized within clinical experience. Institutions should include a role for trainees to partner with QI teams to design regular audits of AI-supported workflows with the goal of aligning systems with existing quality metrics.1,2,6Mesolevel actions represent advocacy inside training programs and institutions. AI literacy and ethics need to be a core competency on par with electronic health record (EHR) training and patient safety.11 Current trainees view AI as an important tool but feel underprepared for its adoption. Faculty resistance and gaps in technological infrastructure remain obstacles to institutional implementation.12 Despite these challenges, all clinicians will care for patients who have received treatment through a care pathway informed by AI clinical decision support tools in the coming years. Without foundational education on AI in health care, trainees will be unable to identify potential pitfalls in these systems.13Examples of curriculum integration include AI-related cases in morning reports and morbidity and mortality conferences, journal club reviews of AI studies with an explicit equity lens, and facilitated debriefs when institutional system inequities are identified. The delivery of such curricula should be multidisciplinary in nature and include perspectives of patient-facing individuals, technology developers, and big data representatives to both inform trainees and promote curiosity for potential future directions.Institutions should also include trainees in their systems for AI governance. There must be advocacy for reserved seats for trainees on AI or digital health committees.1 Institutions should consider designating a trainee “AI liaison,” similar to the “EHR superuser” role that already exists at many medical centers. Frequent touchpoints between trainees and institutional stakeholders should support iterative feedback on system improvement. Additionally, it cannot be understated that the EHR associated with a given hospital system will have a tremendous impact on trainees and their formative experiences with AI in the field. As a result, the privatization of varying AI models in the context of each individual EHR must be communicated to trainees, educated on, and closely examined for any equity-based shortcomings.Macrolevel actions represent advocacy in policy and professional organizations. Momentum for AI advocacy is strong among existing medical associations. The American Academy of Pediatrics (AAP), WHO, American Medical Association, and UNICEF all actively engage in discussions regarding AI ethics and governance.9,10,13 The AI in Pediatrics Webinar Series from AAP could support a longitudinal curriculum at no additional cost to training programs.14 Trainees should receive encouragement to join AAP sections and committees focused on AI, informatics, ethics, and equity. Alongside faculty, trainees should participate in resolution submission and policy development regarding safeguards in AI, as well as take the initiative to write op-eds, blog posts, and other pieces that amplify child-centered AI principles in public-facing spaces. Macrolevel advocacy should explicitly center child health equity, ensuring that policy submissions and public writing address the disproportionate harms that poorly governed AI poses for less resourced pediatric populations. Future work should formally document and publish institutional programs for trainee engagement in AI governance so that case examples can be more readily shared and replicated across training programs.Our hope is that Dr Atrix, future Resident of the Year, feels empowered by her institutional systems to report and act on the shortcomings of hospital AI tools. She applies the skills learned in her AI health care curriculum to critically engage with the models she uses to care for her patients in partnership with her mentoring physicians. Collaborating with supervising faculty, she provides real-time feedback on issues she encounters through a standardized reporting system. She works with her institutional QI team to improve the system early and often. Dr Atrix then translates these first-hand experiences into written work, demonstrating the positive impact ethical AI implementation can have on health equity while also discussing the potential that unethical implementation has in exacerbating inequities.AI in pediatrics will not be neutral. It has the potential to dramatically impact the state of health equity for children from less resourced backgrounds. Whether that impact is positive or negative depends on our willingness as clinicians to prioritize patient well-being over the allure of convenience. The implementation of these tools will benefit tremendously from pediatric trainee involvement at all levels of system analysis.