Determining Entrustment and Practice Readiness in Dental Education: Learner Samples

Health professions are adopting Entrustable Professional Activities (EPAs) as a framework to support entrustment and practice readiness decisions. EPAs describe the tasks, or units of work of the profession, bridging competencies to clinical practice [1]. In health professions education, moving toward an entrustment framework using EPAs implies including concepts beyond competence and skills [2]. The University of North Carolina Adams School of Dentistry (UNC-ASOD) has implemented an entrustment framework that included the development of a longitudinal clinical assessment system using EPAs as part of the Advocate Clinician Thinker Curriculum (ACT) [3-5]. With trust being the highest level of clinical competence [6], UNC's initiative aimed to evaluate graduates' competence and trustworthiness in a more robust manner. Having multiple evaluators assess learners multiple times on tasks representing daily activities of a general dentist in varying contexts aims to: (a) increase faculty trust in learners to consistently provide the standard of care in a safe and independent manner; (b) allow for time variability in learners achieving entrustment and practice readiness; and (c) overcome clinical assessment challenges such as generosity error, lack of reproducibility, and assessing learner based on effort, especially when using a traditional singular opportunity to demonstrate competence [7-9]. This framework, proposed by the American Dental Education Association (ADEA) Compendium of Clinical Competency Assessment [10], was embraced by the ASOD faculty and adopted as a part of the ACT curriculum. This manuscript describes structures and processes required for an entrustment framework and explores learner progression and performance. One EPA was selected to illustrate the process, UNC's EPA 1B: Comprehensive Examination, a component of the patient assessment EPA. This manuscript is part of a series of publications related to UNC's clinical assessment evaluation and quality improvement, and received Institutional Review Board exemption (#24-1240). Standardized rubrics, minimum essential experiences, and practice readiness criteria were determined for each EPA by the appropriate discipline (Appendix 1). Minimum essential experiences are the minimum number of experiences learners must have before being evaluated for entrustment and practice readiness. Practice readiness criteria are the number of experiences in which a learner must demonstrate high-quality and independence. Quality of care is evaluated on a four-point scale using rubrics and including critical errors, and the level of independence is assessed using a four-point modified Ottawa Surgical Scale [11]. For select ASOD EPAs other assessments (e.g., standardized patient encounter, OSCE), complementing patient-based assessments are required. It is important to note that all faculty undergo calibration at onboarding, school-wide retreats, departmental faculty meetings, and annual asynchronous modules on the entrustment framework that include using ACE forms and EPA rubrics for instruction and grading. The nature of the calibrations is focused on both discipline instruction content and assessment, encouraging faculty engagement in the curriculum. Various tools were designed to document and monitor learner performance, including the Assessment of Clinical Encounter (ACE) form [4], a Concern Form, and a Learner Improvement Plan. ACE forms are generated for every clinical encounter and include information on visit context, encounter management, domains of competence, and EPAs. The learner submits the form after completing a quantitative and qualitative self-assessment. Learners are asked to review EPAs, domains of competence, and encounter management rubrics guiding expectations before and after the encounter to prepare for the appointment and self-assess accurately. Rubrics are easily accessible to learners and faculty through a mobile-friendly website application and through links in the ACE form. After learner submission, the supervising faculty receives the assessment and rates quality (meets expectations of a graduate, progressing, or below expectations with or without critical error) and independence (learner's performance during the encounter-retrospective, and performance anticipated for the future-prospective), and provides comments to guide learner improvement. The Concern Form is used by faculty or staff to document non-clinical concerns, such as professionalism issues outside the clinical environment. A Learner Improvement Plan is developed to support a learner who is not progressing as expected, based on benchmarks and expectations for graduation. Multiple below expectations assessments, critical errors, or low clinical attendance are examples of underperformance. The informational technology (IT) team programmed the electronic ACE form, related database, and reports. Additional staff developed Tableau dashboards from the resulting data, providing both static and dynamic reports for learners and faculty. Reports for each EPA indicate quantity, quality, and independence of learners' experiences, illustrate progress over time, highlight experiences assessed as below expectations or critical errors, and allow for class-level (for faculty) and learner-level views over specific timeframes. Two teaching committees (first/second year; third/ fourth year) review learner performance and progress, identify concerns, and provide support as necessary. The Student Progress Review Team (SPRT) evaluates learners for entrustment and practice readiness. The SPRT is advisory to the Academic Performance Committee (APC), confirming recommendations made by SPRT and making final decisions on learner progression, reclassification, graduation, or dismissal. Qualitative and quantitative data constituting a learner's portfolio are reviewed by APC as needed in making final decisions. As part of the teaching committee process, clinical education leaders review learner clinical data prior to each meeting. This data is reviewed by first- and second-year course faculty at the end of each semester, and by third- and fourth-year course faculty, preceptors (group practice leaders), and predoctoral directors at the end of each 6-week block following the curriculum structure (Figure 1) and correlating with increased clinical time. Third- and fourth-year learners generate a “block report” at the end of each block, including clinical data for the past six weeks. Learners complete a self-reflection assignment reflecting on progress and challenges while identifying individual learning goals for the upcoming block. Data from block reports and self-reflection assignments are reviewed by each office preceptor in preparation for teaching committee meetings. In addition to these data, teaching committees consider learner behavior in the context of the ARICH framework (agency, reliability, integrity, capability, and humility) [12]. ARICH outlines key behaviors for trust, reflected in UNC's technical standards policy. This formative assessment process allows for early identification of learners requiring additional support and implementation of timely Learner Improvement Plans. The SPRT or entrustment committee meets approximately six times per year and reviews learners in the final year of the program (Figure 1). The committee includes preceptors, predoctoral directors, director of clinical education, and associate dean for curriculum. Following a standardized process, EPA leads review dashboards to identify learners who have met both minimum essential experiences and practice readiness criteria (making them eligible for entrustment and practice readiness). The EPA lead recommends to the SPRT committee whether to award, maintain, delay, or rescind a Statement of Awarded Responsibility (STAR). This recommendation is based on the learner's ability to perform the specific EPA with highest quality and independence and pass any additional complementary assessments for the EPA. When making entrustment decisions, the context (e.g., visit difficulty) of learners' experiences is reviewed, and learner behaviors are considered using the ARICH framework. SPRT members assess whether learner performance to date supports the committee's trust in the learner to perform the EPA with independence, and quality for their patients going forward, translating into a STAR being awarded. If the committee determines that a STAR is not indicated (e.g., recent performance concern), awarding the STAR is delayed. In addition to awarding STARs, the committee is also responsible for evaluating whether a learner's STAR is maintained or rescinded. To maintain a STAR, a learner should demonstrate consistency in providing quality, safe, and independent care after being awarded a STAR. In the entrustment framework, a STAR may be rescinded, and a STAR Remediation Plan developed to address concerns and provide additional support. Following successful completion of the plan and demonstration of the EPA through quality, safe, and independent care, the STAR may be re-awarded to the learner. At any time, learners and faculty can access an individual “Practice Readiness Report” to view learner progress toward minimum essential experiences and practice readiness criteria, as well as the STAR status for each EPA (Appendix 2). ACE forms populate data visualization dashboards (Tableau Software LLC), organized by EPA. For this article, a deidentified dashboard for the comprehensive examination EPA for the Class of 2024 was utilized to depict various learner progression trajectories. Data included a count of experiences and ratings of quality and independence for every clinical encounter where a comprehensive examination was evaluated. Minimum essential experiences necessary before learners are evaluated for STAR eligibility were set at 15 comprehensive examinations, and practice ready criteria (highest quality and independence) were set at five experiences. Examples of learners with varied number of experiences and time to achieve entrustment and practice readiness were used to illustrate the process. Performance was analyzed as a ratio of the number of practice ready experiences divided by the total number of experiences for each learner. Figure 2 displays the performance ratio of all learners on this EPA and breaks them into three groups: the first tertile as steady, second as moderate, and third as rapid performers based on this ratio. Descriptive statistics of data central tendency (means, median) and dispersion (min, max, standard deviation, and confidence intervals) were calculated using Stata/MP 17 for Mac (StataCorp LP). The Class of 2024 included 91 learners performing a total of 2745 comprehensive examinations (D0150 code) evaluated by 113 faculty. Learners' performance data for the comprehensive examination EPA were reviewed and three examples selected, illustrating a range of experiences and time-variable progression towards entrustment and practice readiness (Table 1). The Class of 2024 averaged 30.33 ± 6.95 experiences (median = 30, CI 95% 18–47) with a range of 17–53. Average number of experiences to achieve entrustment and practice readiness for this EPA was 17.71 ± 6.45, (median = 18, CI 95% 6–31), with a range of 5–33. When analyzing class-level data and STAR distribution for this EPA using time as a variable, data showed that 76 learners (84%) were awarded a STAR in comprehensive examination at the first SPRT meeting (summer of third year), 11 learners (12%) in the fall of fourth year, and four in the spring of fourth year. No STARs for this EPA were delayed or rescinded and all were maintained for the Class of 2024. 1-ASSESSMENT 1B-Comprehensive Exam 30.33 ± 6.95 17/53 Figure 3 illustrates three individual learners' performance on this EPA by day, quarter, and year. Figure 3a shows a learner with steady performance having 23 experiences, represented in yellow that did not reach high quality and independence. Following, the learner was assessed with high quality and independence twice, represented in blue. It was at that point, near the end of the last year of the program, that the learner consistently demonstrated practice readiness on this EPA. Figure 3b illustrates a moderate performing learner with eight experiences until demonstrating high quality and independence in three. Following, the learner had a period of intermittent experiences being assessed as high quality and independence, and then consistently demonstrated high quality and independence in nine consecutive experiences during the final year of the program. Figure 3c shows a rapidly progressing learner able to perform two comprehensive examinations with high quality and independence after five experiences, and performing consistently 28 times more until graduation. So far, three classes (2023, 2024, and 2025) have graduated using the entrustment-based framework using EPAs. While no STARs were rescinded for this EPA for the Class of 2024, six STARs were delayed for the Class of 2025. Those delays were due to concerns regarding the ARICH framework (Integrity) and critical errors in encounter management, including time management and documentation. Although no recensions occurred for this particular EPA example, for this class, STARs were rescinded in other EPAs, such as EPA 2A Treatment Outcome Evaluation, 4A Local Anesthesia, 4B Prophylaxis, and 4C Dentate Direct. Recensions have occurred due to recent assessments with below expectations or critical errors in those respective EPAs. Examples of EPA models supporting entrustment decisions have been published across health professions, with limited dentistry-specific implementation data available. This paper outlines the process of using a longitudinal assessment system utilizing EPAs in dental education for making entrustment and practice readiness decisions, using the EPA of comprehensive examination as an example. As shown by various learner performance experiences, this framework utilized data from multiple patient encounters and multiple faculty, allowing for a robust determination of practice readiness and trust in a learner's future performance on the specified task. This rigorous and comprehensive review by a team of faculty (namely the SPRT, an entrustment committee), combined with considerations of the ARICH framework, is consistent with competency-based education and provides a more holistic evaluation of learner performance with shared decision-making. Within the limitations of the present analysis, this approach highlights time variability to practice readiness and entrustment, providing a more realistic portrayal of learner differences in the time it takes to develop quality and independence. In general, a majority of learners for this particular EPA were determined trustworthy (awarded a STAR) earlier in the review process (summer of third year), mirroring that learners have more experience performing this activity than others at this stage of their educational program. While outside the scope of this paper, additional analyses can assist in determining why learners varied in their rate to achieve a STAR for this EPA. Our analyses have shown that learners who need more time than others to reach practice readiness and entrustment (steady performers) exist for other EPAs. For some learners, slowly progressing performance may be specific to certain EPAs and possibly related to other factors, such as their assigned patients and variances in patient treatment needs. Future analysis taking into consideration data clustering within individuals and time series will further enhance our understanding of this framework. The vast data acquired as a part of the process provides opportunities to identify learners that may be struggling or slower paced, whether due to learner performance or patient assignment, earlier in their clinical education. This allows for implementation of supportive interventions as needed. Noteworthy are the resources necessary to actualize implementation of such a robust model, including time to develop the framework, efforts to collect and analyze data, additional staff to develop and maintain data systems, faculty and learner development and calibration, and time necessary to complete assessments and review learner performance. As a part of implementation, we developed an application in-house to accomplish this work despite there being other commercially available platforms used across health professions learner assessment, due to the need for extensive customization and frequent iteration. To support this assessment model, calibration is frequent and ongoing. Faculty are calibrated annually on using the ACE forms, and discipline-specific calibrations occur year-round, focused on instruction and assessment of EPAs, with institutional rubrics providing a shared cognitive map for learners to prepare, self-assess, and for faculty to evaluate learner performance. This work represents the early phase of implementing an entrustment framework that uses EPAs. As part of our quality improvement process, we continue to refine by analyzing processes and outcomes. For example, criteria for entrustment and practice readiness are reviewed annually by the respective disciplines and adjustments made as needed. As a result of our learner performance, we have established benchmarks to evaluate progression for each EPA for each year of the program. In summary, consideration of the use of EPAs in dental education is promising in elevating and actualizing the goals of competency-based education and evaluating the trustworthiness of our graduates. The authors would like to thank the Information Technology Department, Emily Gadziola, and Tanmayee Muppidi for their support. The authors declare no conflicts of interest. XXXX