Providing effective feedback for training: Is there a role for informatics?

Trainees regularly receive feedback on their performance in clinical medicine, and feedback and assessment are critical to clinical training. In general, summary feedback incorporating written and 360 evaluations has been given verbally via limited exchanges between faculty and trainee. Characteristics of high-quality feedback as identified by the STAROS model include Specific, Timely, Actionable recommendations based on Realistic expectations and Observable Skills-based actions.1 Feedback, however, often occurs less frequently than it should. Additionally, it may not even be recognized as feedback due to time constraints in clinical settings, conflicting priorities, insufficient time and privacy for discussions, the difficulty of delivering negative feedback, power imbalances, and potential for misunderstandings. While evaluative and personalized conversations between trainees and faculty will always be a vital component of feedback, data can also offer valuable insights into a trainee's performance. In procedural fields, including pediatric gastroenterology, data on the types and numbers of procedures performed by trainees are often collected as key metrics for assessing competency. According to the North American Society For Pediatric Gastroenterology, Hepatology & Nutrition training guidelines,2 minimum threshold numbers for competency for esophagogastroduodenoscopy and ileocolonoscopy are 100 and 120 completed procedures, respectively. More recently, the Pediatric Endoscopy Quality Improvement Network (PEnQuIN) identified quality indicators for pediatric endoscopy,3 including adverse event rate, ileal and cecal intubation rates, time-to-ileum, time-to-cecum, and total procedural time. Two academic pediatric institutions, including our own, have assessed the adverse event rates and total procedural times for combined esophagogastroduodenoscopy and ileocolonoscopy, both with and without trainee involvement.4, 5 Adverse event rates remained consistent regardless of trainee participation, while total procedural times were longer when trainees were involved but decreased with each subsequent year of training. Further, evaluation of PEnQuIN quality indicators demonstrated no change in the ileal and cecal intubation rates comparing procedures with and without trainees.5 In contrast, the time-to-ileum and time-to-cecum times were significantly longer for procedures performed with a trainee compared to those performed by faculty alone. Finally, as modern healthcare delivery takes place within electronic medical record (EMR) systems, process data are commonly collected. In Epic, one of the most widely used EMR systems, Signal is a web application that tracks how users interact with the EMR, including how users manage their inboxes, place orders, write notes, and review charts. Signal data have often been used to measure and track user efficiency. In addition, Signal data may also identify significant work burdens and clinicians who are suffering from burnout and who need help and resources.6 A recent evaluation identified patient call messages as significantly associated with clinician burnout. Prior research has shown that pediatric and academic gastroenterologists spend more time at each patient encounter in the EMR than their respective counterparts.7 Data regarding trainee efficiency and performance in the EMR are lacking. The EMR can provide a rich source of educational data.8 EMR data often contain information that has been used to measure quality of care, including length of inpatient stays, medication error rates, and targeted or ideal outcomes, which can be used to determine how well patients are managed by trainees. Further, such data can be opportunistic without requiring additional investment for data collection. In one 5-year emergency medicine program, personalized report cards were given to each resident, reporting metrics including the number of patients seen per hour, bounce-back rate, and time-to-antibiotics and time-to-fluids for patients with sepsis. A follow-up evaluation of resident reactions to personalized EMR data and the value of such data demonstrated that residents appreciated the objective nature of EMR data and viewed the data as valid and complementary to the usual qualitative nature of feedback discussions with faculty.9 A visual summary of the fellow's continuity patients by diagnosis. Figure 1A displays both the volume of patients and the types of patients (categorized by diagnosis) that a given fellow has followed during their 3-year training period. A visual summary of the fellow's in-basket message load. Figure 1B displays the in-basket message volume and type of messages (Staff messages, Results, Patient medical advice, Open encounters, and Therapy plan reminder requests) received by a given fellow over the past 3 months. To supplement Signal data regarding trainee efficiency and performance in the EMR, we have incorporated this in-basket measure and particularly patient call messages (patient medical advice) into our fellow data dashboard as a measure of work burden and potential risk for burnout. Fellow performance with regard to pediatric endoscopy quality metrics (including time to ileum, time to cecum, and ileal and cecal intubation rates). Figure 2 displays mean time-to-ileum over the past 6 months individually for fellows (faculty data are also available separately). Listing of all completed procedures per fellow. Figure 1 data visualizations were created using SlicerDicer, an Epic EMR user-driven analytics tool that does not require informatics expertise. Typically, users request access to or setup for a personalized data dashboard, after which they can create data visualizations through SlicerDicer. Prerequisites include an already-established data infrastructure to identify fellow-attributable data. At our center, continuity clinic patients are scheduled under fellows, and visit diagnoses must be appropriately documented for accurate patient panel display. Prerequisites for generating Figure 2 visualizations have been detailed in previous work.10 For users of other EMRs, access to self-analytic software may be limited and additional informatics expertise required, but the basic data infrastructure requirements still apply. Fellows also have access to their Epic Signal metrics, which provide information on physician efficiency. Sample metrics include time per day spent in in-basket, orders, notes, clinical review, as well as time spent outside of 7 a.m. to 7 p.m. and percentage of appointments closed the same day. Finally, for select patient groups, targeted patient outcome metrics are measured across a given trainee's patient cohort in a disease-specific dashboard. For example, for inflammatory bowel disease, metrics including prednisone-free remission rate, hospitalization rate, emergency room utilization rate, as well as proportion of eligible patients receiving transition readiness services, are displayed for both the entire division's patient cohort and for each individual provider/trainee's patient cohort. Data from the fellow-specific dashboard, Epic Signal website, and disease-specific dashboards can be used for evaluation purposes and also to launch periodic feedback discussions. Incorporation of generated EMR displays into program director fellow feedback sessions has already proven valuable. While most metrics speak to performance and achievement, other metrics (i.e., Signal and in-basket message load data) can be and are used to monitor for work burden and risk for burnout. Moreover, all data are collected in real time, allowing for the evaluation of fellow-specific interventions and their effect on relevant metrics over time. Finally, visual displays can be readily transformed into extractable reports that fellows can use for future credentialing purposes at the end of their training. Trainee feedback is evolving in the modern era, and as a field, we must rise to the challenges presented by practicing medicine within the EMR. Despite its difficulties, the EMR also provides valuable opportunities for trainee feedback and education. While we have highlighted opportunities for trainee feedback that include not only personal performance metrics (procedural quality indicators and Signal data) but also patient outcomes (e.g., prednisone-free remission rate for inflammatory bowel disease), there remain opportunities to provide feedback to fellows regarding their specific role in the delivery of quality care. Burk-Rafel et al. have proposed a conceptual framework for identifiable TRACER (Trainee Attributable & Automatable Care Evaluations in Real-time) measures that are meaningful and scalable11 within competency-based medical education. This framework acknowledges and promotes the role of individual trainees in patient care for specific health scenarios, such as ordering or adjusting medication dosages to maintain or regain disease remission or ordering necessary tests to diagnose a condition for timely intervention. We plan to incorporate some of these TRACER metrics into our own trainee-specific data dashboards in the near future. Foreseeable benefits of incorporation of TRACER metrics include linkages of these metrics to milestone evaluations and ultimately back to entrustable professional activities, as have been previously published for pediatric gastroenterology, hepatology, and nutrition.12 Feedback for trainees is an integral part of their education and growth as clinicians. As the field has shifted more toward EMR-based tools for patient care, there lies the opportunity to use some of these tools to offer objective feedback to trainees regarding their clinical practices in real time. While this would not be intended to replace formative personal verbal and written feedback from faculty mentors, it would add objective data that the trainees themselves can track over time as they build clinical acumen and expertise. As clinical practices have evolved over the past decades with the introduction of EMR, so must our ways of providing feedback to help keep our trainees engaged and prepared for the future of medicine. The authors declare no conflicts of interest.