Development and Pilot Evaluation of the ECALM Program: A Bedside Clinical Assessment Tool for First-Year Residents in the Emergency Department

Siiri Astudillo Kunnas,1,2 Lilia Rotari,1 Farès Moustafa,3,4 Daniel Aiham Ghazali1,2,51Emergency Department, Trauma Center, and EMS, Amiens University Medical Center, Amiens, F-80000, France; 2SSPC UR-UPJV 7518 Clinical Research Unit, Amiens Faculty of Medicine, Jules Verne University of Picardie, Amiens, F-80000, France; 3Emergency Department, Clermont-Ferrand University Medical Center, Clermont-Ferrand, F-63000, France; 4INRAE UNH, University of Clermont Auvergne, Clermont-Ferrand, F-63000, France; 5"Infection, Antimicrobials, Modelling, Evolution” (IAME) Research Unit, INSERM UMR1137, Paris Cité University, Paris, F-75018, FranceCorrespondence: Daniel Aiham Ghazali, Emergency Department, Trauma Center, and EMS, Amiens University Medical Center, 1 rond-point du Professeur Christian Cabrol, Amiens, F-80000, France, Tel +33 322 088 790, Email [email protected]: To describe the development and pilot evaluation of the ECALM program ("bedside clinical evaluation”, in French), a formative bedside assessment framework designed for first-year residents during emergency department (ED) rotations.Methods: The ECALM program was developed prospectively in three stages, in accordance with Stufflebeam’s "context, input, process, product” framework: (i) a baseline needs assessment survey, (ii) the development of standardized clinical scenarios via a structured expert consensus process, and (iii) a single-center pilot implementation. Six core clinical themes were selected (chest pain, abdominal pain, limb trauma, syncope, fever, and confusion), on the basis of epidemiological data from French EDs. A pilot study involving 12 first-year residents was conducted at Amiens University Medical Center’s ED (Amiens, France). Each resident was evaluated with a single scenario, and each session included a structured briefing, passive bedside observation by paired, independent, senior evaluators, and a formative debriefing. Psychometric evaluation of the ECALM tool was based on inter-rater reliability, using the intraclass correlation coefficient (ICC(2,1)) calculated using a two-way random-effects model for absolute agreement based on a single measurement, the weighted kappa, and a Bland–Altman analysis of inter-observer differences.Results: In the needs assessment (physician response rate: 27.1%), 97% of the participating senior practitioners reported that residents were insufficiently prepared for independent clinical tasks. During the pilot phase, the mean ± standard deviation total duration of an ECALM session was 43 ± 12 minutes. A preliminary analysis of the scores’ consistency yielded an ICC(2,1) of 0.76, indicating good inter-rater agreement between independent evaluators. The weighted kappa (0.74) was consistent with the ICC results. A Bland–Altman analysis demonstrated a mean inter-observer difference of only 1.25%, indicating negligible systematic measurement bias between evaluators. Qualitative feedback demonstrated high acceptability: all 12 residents expressed their satisfaction, reported an increased sense of psychological safety, and perceived the session to be a "one-on-one tutorial”. Furthermore, ten residents (83.3%) spontaneously reported acquiring new knowledge during the debriefing phase.Conclusion: The results of this pilot study demonstrated the operational feasibility, high acceptability, and good preliminary inter-rater reliability of the ECALM program for the formative assessment of first-year residents. Although ECALM constitutes a promising, non-punitive approach for encouraging early clinical reasoning and self-assessment, further well-powered, multicenter validation is required to demonstrate the tool’s external validity and long-term impact.Keywords: emergency medicine, formative assessment, clinical supervision, evaluation in healthcare, clinical reasoning, teaching at the bedside, ECALM