Development and validation of Trainee Attributable & Automatable Care Evaluations in Real-Time (TRACERs)

PURPOSE: To develop Trainee Attributable & Automatable Care Evaluations in Real‑Time (TRACERs) for inpatient diabetes management, collect validity evidence for their use in formative assessment, and explore performance variation across residents and institutions. METHOD: In 2023, a multi‑institutional team created two TRACERs based on type 2 diabetes guidelines-discourage bolus‑only insulin (TRACER #1) and encourage basal (± bolus) insulin (TRACER #2)-for internal medicine residents at three large residency programs. Residents were attributed to inpatient admissions based on placing the most medication orders in the first 12 hours. Structured queries extracted 35 discrete variables from the electronic health record (EHR). Two experts per institution reviewed random admissions (July-August 2022) to establish criterion validity. A retrospective cohort (July 2020-June 2023) added validity evidence. RESULTS: Automated extraction achieved ≥96% sensitivity and ≥95% specificity when compared to manual review. Among 615 residents attributed to 6,192 admissions of patients with type 2 diabetes at high risk for hyperglycemia, TRACER #1 occurred in 42.6% (1,689/3,965) of admissions at Program A, 28.9% (408/1,410) at Program B, and 26.7% (218/817) at Program C. TRACER #2 occurred in 44.9% (367/817) of Program C admissions versus 24.2% (959/3,965) at Program A and 28.2% (397/1,410) at Program B (all P < .001). Four resident-level insulin‑ordering profiles were identified-consistent basal-bolus insulin use (most guideline-concordant), basal-predominant, bolus-predominant, and bolus-only (most guideline-discordant)-with between‑resident variation exceeding between‑program differences. Longitudinally, cohort-level trends masked opposing individual trajectories-some residents improved with exposure while others worsened-and performance tertiles were distinguishable early in training. CONCLUSIONS: TRACERs revealed substantial institution‑ and resident‑level variation in insulin‑ordering practices, including guideline deviations, demonstrating potential for real‑time formative feedback. Multi‑institutional implementation highlighted scalability barriers, including EHR heterogeneity and workflow‑dependent attribution, underscoring the need for continued refinement and broader validation.